Manage Your Jira Data Center Instance Wisely With Guardrails - News, tips & guidance for agile, development, Atlassian-Software (JIRA, Confluence, Bitbucket, ...) and Google Cloud

In late June of 2022, Andrzej Kotas, a product manager for Jira Data Center, announced to the Atlassian community an exciting development in documentation and recommendations for Data Center deployments of Atlassian Tools. The post introduced Jira and Confluence guardrails.

Guardrails, as put by the Atlassian team, are “product-based documentation and strategic recommendations designed to educate and help you avoid reaching a tipping point where your instance might start to experience performance impacts.”

You might have read about Confluence Guardrails in our previous post, Manage your Confluence Data Center Instance Wisely with Guardrails. In that post we summarized the new recommendations with Confluence Data Center instances.

Today we are going to do the same thing, but this time we will focus on Atlassian’s flagship product, Jira.

Jira-specific Guardrails

All of the following guardrails are specific to Jira Data Center instances. This can also include Jira Service Desk, depending on your setup.

Projects

For this guardrail, we are referring to the number of active projects. It’s important to distinguish between active and archived. In Jira Data Center, admins have the opportunity to archive projects. For this threshold, we are ignoring archived projects and setting the guardrail at 7000.

Now 7000 projects is a lot of projects, but if your organization has hundreds of thousands of users, it’s likely this could be you. Here’s a link on how to clean up old projects. It mainly consists of you setting a date limit where you say something like “projects inactive for two years get archived”. This document will also help you set a governance policy for other Jira admin related activities.

If your instance goes above this project limit, the number of active projects could make permission calculation complex and therefore slower. Another potential drawback is you could lose the ability to create new projects, as the application will ‘hang’ for 20-50 seconds per active node and could timeout. Finally, reindexing your project will take some time.

In order to avoid these drawbacks, please archive inactive projects.

Comments

For comments, we are focusing on the number of comments per issue. Even more specifically, for this guardrail, we are looking for the threshold of 1000 comments per issue. This isn’t an average number of comments, but if any issue has more than 1000 comments, your instance could be at risk.

Much like our project threshold, 1000 comments is a lot and typically happens when there is a bug or error in your instance. However, here’s a handy link on how to find the issues in your instance with the most comments.

If you are operating above this threshold, the issue view could load very slowly. In extreme cases, there are out of memory errors and that could cause an application crash. Finally, reindexing could take a very long time.

In order to mitigate this guardrail, you could limit the number of items a group's members can create using Safeguards. Another option is to remove old comments using the REST API or Scriptrunner. One of the main causes for too many comments could be a rogue automation rule which is adding unnecessary comments. Take a look through your automation rules to see if you have any unnecessary rules. Finally, you could implement rate limiting to prevent a misconfigured integration from adding thousands of comments in a short time.

Attachments

Here, we are focusing on the number of attachments per issue. The limit is the same as comments for this guardrail. We want our issues to have fewer than 1000 attachments. Additionally, we would like the attachments to be less than 10 MB per single attachment.

If you want to find the issues with the most attachments in your instance, Atlassian has some documentation to help you out. Depending on your database system, Atlassian recommends different queries. If you decide to test some of these queries out, be sure to test them on a staging instance first.

If you are operating an instance that has an issue with over 1000 attachments, you could experience a very slow issue view screen loading. You could also see excessive load on the shared home filesystem. Finally, some issue operations could trigger serializing the issue with all issue properties, including all issue attachment properties.

If you want to avoid these drawbacks, reduce the attachment size limit. You should also check automation rules to see if any attachments are added unnecessarily. You can also turn off the thumbnail display for attachments and archive attachments in projects that aren’t archived already.

Issue links

For issue links, we are talking about all issue links. This includes the links from one issue to another and links to sub-issues. It’s also important to clarify here that "sub-issues" is not limited to just subtasks, but for all issue types below a parent issue. If the main issue is an epic, this could be stories, tasks, or even bugs. The guardrail here is 1000 issue links.

Here is a link to Atlassian’s documentation on how to find the issues in your instance which have the most issue links.

If you are operating an instance that has issues with over 1000 issue links, you may experience slow load time for the issue detail view. You also can get some stuck threads, which can have an impact on the whole instance.

If you experience any of these symptoms, Atlassian recommends you to identify issues that have too many issue links and remove the unnecessary ones. Using the queries provided in the link above, you should be able to identify the issues. From there, contact the issue or project owner to get buy-in on removing the issue links.

Text

Text inputs are important for issue tracking. Every issue requires a summary, so at the very least there needs to be some text on every issue. For this guardrail, we are particularly concerned with the single and multi-line text input fields. For single-line fields, the guardrail is 255 characters. For multi-line fields, the guardrail is 10,000 characters.

If you are operating above these thresholds, Atlassian says you can expect increased index size and slow loading search results.

In order to mitigate this issue, you should consider setting a character limit for text fields. Use this link to find out exactly how to do that.

Custom fields

Any experienced Jira admin knows that custom fields can be a massive pain if not managed regularly. This guardrail is related to the total number of custom fields in your instance. The guardrail is set at 1200 custom fields. While this is a lot, it’s common for us to see instances quickly approaching this number. We believe a good governance system can help to keep this number in check and it’s better to start that governance system early in the lifecycle of your system.

Atlassian has some recommendations on how you can analyze the usage of custom fields in your system. Most of what can be done here can be done through the UI, which makes the task even simpler.

If you are operating an instance with over 1200 custom fields, you may experience overall performance degradation and reindexing could take a long time.

There are two things you can do to avoid these pitfalls in your instance. The first is to analyze custom fields using the link above. The second thing you can do is to optimize your custom fields. Optimizing will provide you with some recommendations on how to get more out of your custom fields. Some fields might have a global context, meaning they are applied to all Jira projects, when in fact only a few projects are actually using them. This is one way to optimize your custom fields.

Epics

This guardrail is straightforward. We are looking at the total number of Epics in the system. The threshold here is 100,000 epics. That’s a ton of epics!

In order to identify the total number of epics in your system, you should use JQL. A simple query like “issuetype = epic” should do it.

If your instance is operating with over 100,000 epics, the epic link menu could load slowly.

The only way to mitigate this potential drawback is by archiving epics. Here is a link with documentation on how to archive issues.

Sprints

For this guardrail we are looking at the total number of sprints in the system. This can be closed, open, and active sprints. The threshold is 60,000 sprints. Again, this is a lot of sprints, but if you have an instance that is ten years old and you are running sprints in thousands of projects, you can easily hit this number.

In order to find the total number of sprints in the database, there are a few different queries that you can use depending on your database.

If your instance is hovering around 60,000 sprints or above, you may experience overall performance degradation due to slow sprint cache population. You may also see that the “closedsprints()” JQL function does not work.

If you want to avoid these drawbacks, you can delete closed and old sprints that are no longer needed. There is a REST API operation which will allow you to do this quickly and easily. Something else you can do is to set the jira.search.maxclauses system property to decrease the default limit to less than 65,000.

Workflow scheme bulk actions

For this guardrail, we are particularly interested in the bulk operation feature of Jira. The guardrail is a threshold for how many issues you can bulk update at once. In this case, the limitation is also 1000 issues. This is actually built in to all Jira products, if you need to perform a bulk operation through the Jira UI, whether you are on Jira Cloud or Jira Data Center, you can only update the workflow on 1000 issues at a time.

If you find a way to operate above this guardrail, you could see bulk operations taking an extremely long time, sometimes up to days. In other cases, you might not be able to see the progress of workflow modification without shortening the URL.

In order to avoid these drawbacks, you need to make a change to your issue’s workflow follow these steps. First, copy the workflow scheme that you would like to edit, make the changes you would like to the workflow, and reapply the workflow to each project one by one.

The other option is to do nothing at all. If you update over 1000 issue’s workflows, you aren’t putting any stress on the system itself and you won’t experience any performance degradation. It will just take a very long time. The only important point is to make sure to not restart your Jira instance during this process.

Change history

For this guardrail, we are focusing on the number of changeitems or changegroups associated with an issue. The threshold here is 20,000 changeitems or changegroups.

In order to retrieve issue change history, follow these instructions.

If an issue has over 20,000 changeitems or changegroups, you could experience out-of-memory errors when viewing the History tab. You could also find issue view and other actions loading slowly. Finally, reindexing could take a long time.

In order to mitigate this issue, use a database query to identify issues with large changeitems or changegroups. Next, clone the issue. The history will not be copied to the new issue.

Users

This is the first Jira guardrail that is similar to Confluence. Here we are referring to the number of users synchronized between LDAP and Jira. The guardrail is 100,000 users.

In order to find the total number of users in Jira Data Center you can use queries from this knowledge base article.

If you are syncing over 100,000 users between LDAP and Jira, you could experience instance instability. This could be as extreme as outages and noticeable performance drops under a heavy load. You could also see an increased time for directory synchronization and user authentication.

If the majority of your user accounts in your instance are stored in Crowd Data Center, Crowd Server, or Microsoft Active directory, you should enable incremental synchronization. This is a setting so only the changes since the last synchronization will be queried. Another option, if the prior mitigation option is not available to you, is to consider using Crowd Server and Data Center as your external user directory. These tools use access-based synchronization. Finally, using LDAP filters can help reduce the number of users and groups to process in your instance.

Inactive Users

For this guardrail, Atlassian are referring to the number of inactive users with content assigned to them synchronized between LDAP and Jira. The threshold is 100,000 users.

If your site ends up exceeding the 100,000 inactive user amount you could potentially experience instance instability with noticeable performance drops under a heavy load. You could also experience increased time for directory synchronization and user authentication. Additionally, if you are running a version of Jira which is older than 8.20.6 the guardrail is actually closer to 10,000 inactive users with content assigned to them.

To mitigate this guardrail, first look to upgrade your Data Center instance to 8.20.7 or later. If you can’t upgrade at this time, unassign the inactive users from Jira content, and then run a sync to remove inactive users from your directory.

Groups

Here, similarly to Confluence, we are focusing on the number of groups synchronized between LDAP and Jira. The guardrail here is 25,000 groups. This is a lot of groups, but in our experience some enterprise organizations have this many groups and potentially more.

If your instance has above 25,000 groups, you could potentially experience instance instability including outages and poor performance. You could also see an increased time for directory synchronization and user authentication. Finally, the application access and group management UI could be unresponsive.

In order to mitigate this guardrail, you could configure your LDAP connection pool. Too many or too few connections may have a negative impact on performance. You could also disable group sync on every login by changing the Update group membership when logging in option to only “for newly added users” or “never”. Lastly, becoming familiar with user management limitations and recommendations will help avoid this threshold.

Depth of Nested Groups

This guardrail is also very similar to the depth of nested groups Confluence guardrail. After all, if you have both Confluence and Jira, it’s typical for your LDAP to sync with Jira and for you to do the bulk of your user management there. If you haven’t read our Confluence Guardrails post, we will repeat ourselves here.

The limit for this guardrail is four layers. Atlassian also recommends that groups do not contain groups and users, rather groups should contain one or the other in order to avoid this guardrail.

When systems are operating above this guardrail, Atlassian has noticed instance instability which includes performance degradation and potential outages when usage in Confluence is at a high end. They also noticed directory syncs can take a long time and user authentication can take longer than expected.

The mitigation options for this guardrail are simple. Avoid having too many layers in your groups. You should also avoid having a mix of users and groups within a group if you do have nested groups. This is something that is handled outside of Confluence and will be up to your AD admin.

Keep your instance in check

Overall, if you make checking these guardrails part of your regular instance maintenance, you should have a good start on keeping performance up in Jira. Every Data Center instance of Jira varies slightly so these guardrails will not cover your instance completely, but they will help to establish a base of checks for you to follow.

If you have any questions about maintaining a Jira Data Center instance, or need help setting up a governance program for your Jira instance within your organization, please contact us at Seibert Media.