Skip to main content

Batch Data Calibration

Batch data calibration is a powerful feature that can significantly improve the efficiency and accuracy of data management, including:

  • Refreshing field values.
  • Refreshing options sorting or values assigned to options.
  • Refreshing encryption rules.

Access to batch data refresh: In the table view, users can enable batch refresh by selecting all or multiple records.

This feature is only available to application administrators.

1. Why to refresh field values​

When dealing with large worksheets, especially when the number of records exceeds 100,000 rows, optimizing system performance becomes particularly important. In order to maintain smooth operation and efficient response of the worksheets, the system does not automatically refresh historical data when modifying field configurations (such as adjusting calculation formulas in formula fields). This design is aimed at avoiding excessive system load caused by large-scale data updates, which could negatively impact user experience.

For example, let's say you modified a formula field. Initially, the calculation formula for this field was "Quantity x Unit Price." After some time of use, you may need to adjust the calculation formula to "Quantity x Unit Price x Discount" based on business requirements. If the number of records in the worksheet is less than 100,000 rows, the system will automatically refresh existing record data after you save the form. However, when the number of records exceeds 100,000 rows, for performance reasons, the system will not automatically perform this update, requiring you to manually refresh the data.

Controls that require manual refresh include Formula, Rollup, Concat and Foreign field.

2. Why to refresh options sorting​

The logic behind option sorting is that each option is assigned a hidden sequence number, which is used to uniquely identify and record the position of the option in the list. When we select an option in a record, the system records the sequence number of that option. Subsequently, when there is a need to sort records based on options, the system sorts them based on these sequence numbers.

However, in practical applications, we may need to adjust the position of options or add new options based on business requirements. These actions will result in the reassignment of option sequence numbers. For example, if there were originally only options A, B, and C with sequence numbers 1, 2, and 3 respectively, and we then add options A+, B+, and C+, the sequence numbers of the existing options will change. In this example, the option A, originally with sequence number 1, may be changed to 2 or another value depending on the position where the new options are inserted.

After option positions change, the sequence numbers stored in historical records will not be automatically updated and will require manual refresh.

3. Why to refresh values assigned to options​

The mechanism for updating the values assigned to options is similar to the updating of options sorting. When the value of an option changes, the previously assigned and stored value does not automatically update. This is because historical data is usually considered unchangeable records, similar to the sales price in historical orders, once recorded, they remain unchanged to ensure data authenticity and traceability. Therefore, if we need to update the current value of an option and want this update to be reflected in all related records, a manual refresh operation is required.

Notification after refresh completion​

The refresh operation is executed asynchronously, meaning that users can continue with other tasks while the refresh is in progress without needing to wait for it to finish. However, if the data volume is large, the refresh process may take some time. In order to promptly inform users of the progress and results of the refresh operation, the system sends a message notification after the refresh is completed. This way, users can stay informed of the refresh status and take appropriate actions.

Log after refresh completion​

After the refresh is completed, the system generates worksheet logs. By reviewing these logs, users can clearly understand the specifics of each refresh operation, ensuring data accuracy and traceability.

Minimum 2-hour refresh interval​

In order to prevent frequent refreshes from negatively impacting system performance, the system has set a refresh interval limit. Specifically, after each batch data refresh, users need to wait at least 2 hours before initiating another refresh operation.

Maximum 100,000 rows refreshed per time​

To improve refresh efficiency and reduce system load, the system also imposes a limit on the number of records refreshed at a time. Specifically, a maximum of 100,000 rows can be refreshed per operation. Therefore, when selecting data to refresh, users are advised to use filtering criteria to limit the number of rows, ensuring that the data volume for each refresh is manageable. If the data exceeds 100,000 rows, the system will automatically select the first 100,000 rows based on creation time for refresh. This limitation helps protect system performance, ensuring that the refresh operation can be completed efficiently and stably.