What should we pay attention to when preparing for release, how to release and what to monitor once released.

Continuous Integration and Continuous Deployment (CI/CD)

Set up a CI/CD pipeline to automate the process of building, testing, and deploying your code. This helps in ensuring consistent and reliable releases.

Blue-Green Deployment

One way of minimizing downtime and risk is by maintaining two identical production environments (blue and green). Deploy updates to the inactive environment, test thoroughly, and then switch traffic (little by little if you want) to the updated version.

You can also do this in your apps using feature flags. Maintain the old code while writing all new functionality behind a new flag. When you feel sure there are no bugs or you don’t want to go back - delete the old version.

Canary Releases

Gradually roll out new features or updates to a small subset of users before deploying to the entire user base. This helps catch any issues before they affect all users.

Automated Deployment

Create scripts or use deployment automation tools to ensure consistent and repeatable deployments across different environments.

Security Best Practices

Secure your software by following security best practices like input validation, data encryption, XSS, and other secure coding practices. OWASP is a good source for cyber security.

Scalability and Efficiency

Design your software with scalability in mind. Use efficient algorithms and data structures, and consider horizontal and vertical scaling options.

Release Early, Release Often

Smaller pull requests lowers the risk of introducing errors and increases development speed.

Hide unfinished work behind feature flags and ship internally as soon as we can and iterate on solution.

Use canary releases (beta/staging), blue-green deployment or remote configuration to give access to stakeholders, staff or external testers as early as possible (shifting left again) to help us find bugs or issues. Once we get our hands on our work we could discover better solutions and want to be agile and change our direction. People only know what they need from the software once they’ve had a chance to use it.

In general, avoid big releases, they are much harder to review and test and we want to ship quality software.

Feature flags / Remote Config

Use remote configuration or release flags liberally. The goal should be to always release changes behind a flag to allow quick “rollback”.

Ideally every real app release has no visible change from the previous version. Instead, releasing a new feature means you enable the flag in your remote config system. This decouples feature releases from app-version releases, and allows you to manually control who and when something is actually released to the public (and do release internally first!).

Having a feature flag system which allows targeting specific releases, users and user segments (like all employees) is important. This allows you to easily release beta functionality to testers, or slowly roll out a new feature.

Rollback Plan

Have a well-defined rollback plan in case something goes wrong during deployment. This includes reverting to the previous version and ensuring data integrity.

Database Migrations

Plan and execute database migrations carefully. Use versioning and scripts to manage changes to the database schema, and always back up data before making changes.

Decide if app code should be deployed before or after running migrations and how your app will handle a new schema during migration - do you stop/pause the services?

Ideally make database changes reversible and backward compatible (with app code) and have a plan for handling it when this is impossible or too expensive.

Avoid complicated or slow statements while doing migrations, or schedule downtime for them.

Use an ORM when possible to avoid typos and making statements simple and easy to read.

Load Balancing and Scaling

Implement load balancing to distribute traffic evenly across multiple servers. Use auto-scaling to dynamically adjust resources based on traffic demand.

Scheduled Maintenance and Downtime

Communicate planned maintenance or downtime to users well in advance. Decide on a standard for what that means to your services.

Use maintenance windows during off-peak hours to minimize disruptions.

Have systems in place, like banners or alerts, for informing people using your service before and during downtime.

Force Updates and Nudge Systems

Sometimes we make mistakes which are irreversible, or it’s finally time to shutdown an old and deprecated api. When this happens we need ways to force or notify clients to upgrade.

Forcing is always a harsh method, especially towards end users, for this reason having a nudge (non-forcing) information to upgrade clients is useful.

Ironically this is a feature that typically gets built way too late, after it was already needed, so don’t forget to add it in early!

Content Delivery Networks (CDNs)

Utilize CDNs to cache and serve static content from servers located closer to end-users. This improves performance and reduces the load on your servers.

Security Scans and Audits

Conduct security scans and audits before and after deployment to identify vulnerabilities or compliance issues. This includes code reviews, penetration testing, and vulnerability assessments. There are good tools to help us with this, make sure to research what applies to your platform, language, framework etc.

Immutable Infrastructure

Treat infrastructure as code and use images or containers that are replaced with each deployment, rather than making changes to existing instances.

Incident Response Plan

Have a well-defined incident response plan in place to address any unforeseen issues or outages promptly. This plan should include communication protocols and escalation procedures.