Git Integration for Jira Self-Managed (Data Center/Server) Documentation

Multiple Indexing Threads – General setting

NEW FEATURE ADMINS DATA CENTER ONLY

Available in version Git Integration for Jira Data Center v4.27+.

Many git administrators express concerns regarding the performance issues associated with indexing repositories. Our current single-thread indexer has been found inadequate for effectively addressing this situation.

Starting GIJ version 4.72+, an extended feature has been introduced to support the use of multiple indexing threads for each node defined by the user for indexing. This improvement aims to greatly enhance the efficiency of the indexing process.

This feature allows the configuration of multiple indexing threads per node in the General Settings page. This new setting is introduced to consolidate all indexing threads from across all nodes.

Enter the required number of indexing threads per node. The default value is 1; maximum value is 100. This is the number of indexing threads available for updating repositories from remote git servers. Please note that this feature is experimental. Make sure to adjust this setting accordingly while taking into account the consequences if the hardware requirements do not align with the number of indexing threads.

Multi-threaded indexing may cause a slight delay for API responses.

Important
Additional indexing threads will require additional CPU and memory resources.

Rate limit throttling settings

HTTP code 429 (too many requests) is raised if the number of requests overwhelms the limit of the git service. Take into consideration that processing too many requests consumes more CPU and memory resources.

To avoid problems with excessive load on external git services, we introduced rate limit throttling. This feature allows administrators to configure the number of requests to external git services measured in requests per second (rps). Setting the value to zero (0) for the selected service will disable throttling for that git service.

For integrations from private git servers — this group setting is disabled by default. This is because private servers can be tailored to the organization’s requirements and policies.

For integrations from .com (cloud hosted) — this group setting is enabled by default. Make adjustments according to the cloud-hosted rules and limits.

Fix rate limit errors by reducing the number of requests to your git service. The above throttling settings allow administrators to configure the rate limit. This ensures that the Git server is not overloaded, especially with very large git integrations with the Jira instance.

The GIJ app is designed to efficiently receive and process all necessary data, regardless of any potential rate limits in place. It prioritizes maintaining the integrity and security of the data at every stage to avoid any risks of loss or corruption. Note that commits indexing will still work regardless of rate limit errors.

Refer to the table below as a basis for rate limiting:

Git service	Rate limit
GitHub.com and GitHub Enterprise Cloud	5000 requests per hour per user
GitHub App	5000 requests per hour per user Can be more, depending on conditions for user authentication and membership in the organization
GitHub Enterprise Server	Default: *disabled* Depends on git admin setup or hardware resource limit
GitHub App Enterprise Server	Default: *disabled* Depends on git admin setup or hardware resource limit
GitLab	300 requests per minute
GitLab CE/EE	Default: *disabled* Depends on git admin setup or hardware resource limit
Microsoft Cloud	200 requests per 5 minutes
Microsoft On-premises	Default: *disabled* Depends on git admin setup or hardware resource limit

Repository hit rates

Track hit rates by reviewing current and maximum sizes in MB (megabytes):

Revision cache
Branch cache
Tag cache size

GitHub rate limit throttling

After learning about GitHub’s primary limits, GitHub also imposes secondary rate limits. These restrictions on concurrent calls from systems should be no more than…

100 concurrent requests allowed.
900 read requests per minute or 180 write requests per minute to a single REST endpoint.

The rate limit for GitHub Apps depends on whether the app authenticates with a user access token or an installation access token. It also depends on where the app is owned by or installed on a GitHub Enterprise Cloud organization.

For more information, see the following articles:

GitLab rate limit throttling

When the limit for the number of concurrent requests to GitLab is reached, a 429 error is returned. To perform another attempt, the client should wait for a bit. To learn more about GitLab rate limits, see GitLab.com-specific rate limits.

Microsoft rate limit throttling

Limit varies based on the pricing tier (e.g., Free, Shared, Basic, Standard, Premium)

Some Azure services have adjustable limits. The limit can be raised above the default limit but not above the maximum limit. Some limits are managed at a regional level. Note that free trial subscriptions are not eligible for limit increases.

For more information on Azure rate limits, see Azure subscription and service limits, quotas and constraints.

Factors influencing performance

Some factors that affects reindex performance:

number of branches
number of commits in a branch
number of git notes in the repositories
number of commit issue associations
commit notification emailing is enabled
smart commit is enabled
script execution is set up
the setting “Allow new commits to change LastUpdated field” is enabled
reindex queue size
reindexer idle time

Some factors that affect the Jira issue page loading performance:

Repository browser enabled
tags are displayed
number of tags and tag chains
number of repositories associated with a project
number of disabled repositories
the global setting “Enforce Git service permissions” is enabled

For Webhook performance, check the…

value of “Minimum repository reindex interval”
max. number of webhooks per hour from when you first use GIJ app
number of webhooks for the last hour or day
number of reindex tasks added by the webhook event

Optimizing repository indexing threads

The indexer has the capability to execute from 1 up to 100 repository indexing tasks concurrently on each node. The amount of threads can be configured on the “General settings” page under the “Number of indexing threads per node” field. Setting the right number of indexing threads is crucial to avoid surpassing your git service provider’s rate limits.

We recommend to start with the default number of indexing threads. Incrementally increase the thread count to monitor indexing speed improvements while observing for rate limit issues or performance decline. This approach will help assess the performance impact and ensure stability before making further adjustments.