Regression Testing

Introduction to Regression Testing with Devin

Embracing Regression Testing

In the dynamic world of software development, ensuring the reliability and functionality of your application through updates and enhancements is important. This is where regression testing comes into play, a concept that, though widely recognized in general software development, is relatively new within the Claris FileMaker community. Regression testing involves systematically checking that recent scripts or schema changes haven’t affected or broken existing features, ensuring that your application continues to operate as expected after shipping new versions to your clients.

Why Regression Testing Matters

• Maintain Integrity: As your FileMaker project evolves, regression testing helps maintain the integrity of existing functionalities, ensuring new changes don't introduce errors.

• Enhance Quality: It's a proactive measure to enhance the overall quality of your application, contributing to a more stable and reliable software product.

• Boost Confidence: Knowing that changes won't break existing functionalities boosts confidence among developers and stakeholders regarding the continuous improvement of the application.

Testing Methodology

With regression testing, we want to validate both individual components ("units") and their interactions with external dependencies ("integrations"). This ensures your application's robustness before releasing changes to a branch or before updates are deployed between environments.

Unit Tests

These tests focus on individual components or functions within your application, isolating each piece to verify its correctness. An example could be testing a script that calculates the total price in an invoice. By writing a Test Script that feeds predefined inputs into this function and checks if the output matches expectations, developers can ensure that even the smallest units of the application perform correctly.

Integration Tests

While unit tests examine the pieces, integration tests check how those pieces work together. An example might involve testing the entire invoicing process, from adding line items to generating the final invoice. Another example could be to validate that you get an expected http code back from an API, given a certain json payload. This ensures that when individual components interact, they do so as expected, and the end-to-end process is error-free.

Implementing Regression Testing using Devin

Devin introduces a simple testing framework that seamlessly integrates with your FileMaker Script Workspace, allowing you to write Test Scripts.

• Organized Test Scripts: In your file’s script workspace, create a folder for your tests. By default, Devin will look for a Tests folder, but you can specify a custom name in the Project Settings. Devin will interpret all scripts in this and any sub-folders as test scripts.

• Defining Success: Utilize the Exit Script [True] command within your test scripts to signify success. If a test script does not return True, Devin will interpret the test as FAILED.

Example of a Script Workspace with three Test Scripts

Then, access the Modules -> Tests tab in Devin's Project Workspace to Load Test Scripts and Run Test Scripts.

Example of Tests running for a Module on a dev Branch

Devin utilizes the FileMaker OData API to execute your tests. For successful test execution, ensure that the odata extended privileges are enabled for the account specified when adding the Module to Devin. Additionally, be aware that the FM OData API restricts script names from starting with numbers or containing special characters.