- Abstract
- Create a Test
- Working with Files
- Advanced Test Properties
- Master/Slave Setup
- Advanced JMeter Configuration
- Customizing JTL Properties
- Selenium for User-Experience Monitoring
- 1st Time Use
- Launch a Test
- Accessing JMeter Console
- Running Your Load Test
- Restarting a Test
- Monitoring the JMeterEngine(s) Vital Signs
- JTL files and Real Time Reports
- Working with Test Results Generated by JMeter
- Saving Load Test Results
- Accessing Files that were Created During the Test
- Using CSV Data Set Config in the Cloud
- How to setup different CSV file for each JMeterEngine
- Recording HTTP traffic in the Cloud
- Best Practice using PXPro
JMeter for the load, Selenium for user-experience monitoring.
PerformanceXpert professional edition, designed for professional use, is equipped with a self-service platform and advanced scripting capabilities leveraging JMeter and Selenium.
You can create proprietary test scripts and load scenarios using a graphical web environment. PXpro offers web-based test management, archiving, repository, cloud-based monitoring, rich scripting language, and supports HTTP, HTTPS, web-services, XML, TCP, SQL, Login and more.
PXpro enables you to write load test-scripts using JMeter and user-experience test-scripts using Selenium. PXpro will generate a load based on the JMeter script. The Selenium script is used during the load to automate the launch of real browsers to measure the real end-user experience.
The load and monitoring is using a pre-configured distributed load testing environment. The environment is ready to use and available at all times.
If you are not familiar with Selenium and do not wish to create a Selenium test-script, the system can generate one for your programmatically based on landing pages you provide. If you want the same for your JMeter script, PXlite generates JMeter scripts automatically.
With PXpro, all you need is to write the test-scripts, choose the amount of load-engines and run the test. The system takes care of the everything else. Unlimited number of load-engines are pre-configured and available at your disposal. Detailed graphical reports are generated during the load.
Add Test
Use the Add Test menu function to create a new test. Follow the instructions on the page to complete the required details.
Using the GUI Vs Running Headless
PXpro enables you to run JMeter in two different configurations:
- Use the console's graphical user interface
- Run headless without any console
Running headless, without the console, will start JMeter immediately providing it with the uploaded test script (i.e. my_script.jmx). This configuration is highly efficient in resource allocation especially in terms of memory. You can use more engines and a more intensive scripts using a headless configuration with less risk of encountering memory related issues with JMeter.
You can use the JMeter GUI to edit your script, debug it and run it via the console. Through the console you can also enjoy inherent reports available only within the JMeter console.
JMeterEngines
The load is generated by a distributed JMeter architecture. A JMeter console is used to control the test. You need to choose the number of JMeterEngine(s) that will participate in the test.
We recommend running up to 300 threads/virtual users per one JMeterEngine if you are using HTTP/S protocol. If you are using any other protocols such as Web Services we recommend up to 70 threads/virtual users.
The number of supported threads depends on your script intensity and available resources. The more intense your script – the less threads a single JMeterEngine can support. For example, a test with 10 JMeterEngine(s) each running 300 threads, simulates a total of 3,000 users.
In general, we recommend not to go over the following limits:
- up to 300 hits per second per JMeterEngine
- Up to 100Mbps per JMeterEngine
*** We highly recommend running a high number of JMeterEngine(s) using a headless configuration. A headless configuration is much more efficient in terms of resource allocation. ***
Load Geographical Origin
You can choose the specific geographic location from which you wish to run your load. This selection can be modified before each run, that way you can run the same test several times and each time choose a different location in the world. We provide four global locations, two in the US, and one in Europe and one in Asia.
Upload your JMeter Test Script
You can optionally upload your JMeter test script. Once uploaded, you can perform the following actions on the uploaded file:
- You can edit this file at all times using the JMeter console provided for you once the servers are launched running in GUI mode.
- You can use the Files tab to upload additional files such as CSV or TXT files. The Files tab can also be used to upload additional JMX files. Working with files is also available during the actual test (i.e. editing, downloading and uploading files) using the Files tab.
Save Your Test
Don't forget to save your test.
Unlimited Load Capacity
For unlimited load capacity, launch as many tests as required in parallel. For example, to generate a load of 12,000 users you can launch three tests of 4,000 users.
Upload any file you wish to use during the load test.
The Files tab will appear only after you save the test for the first time.
At that point:
- The Files tab is accessible at all times. You can use it to upload, download and delete files.
- All files in your account are downloaded to the remote JMeter servers at the beginning of each test.
- During the test, files in your account are synchronized every few minutes.
- You cannot delete files during the load test. Files can be deleted only once the test has ended or before it has started.
- Prior to terminating your test session, please make sure all files are synchronized.
- Do not use this tab to upload the Selenium test script. There is a special place for this.
Limitations on File Names
- File names should include only alphanumeric characters, underscores or hyphens: [0-9],[aA-zZ],[_-].
- File names should always include an extension.
For example:
- A file name: "my_file_no_4.csv" - is good.
- A file name: "m$ file &" - is bad.
Affecting Speed of the Test
All files are being downloaded to all of the remote servers each and every time a test starts.
A test will not start and the JMeter console will not be accessible before all of the files are downloaded.
For example, if the Files tab contains numerous large files, it can take few minutes for them to download and the test can not start prior.
Therefore, we recommend not to have files that are not in use with-in the Files tab. It is best to clean this tab prior to launching a test.
In the Add/Edit Test page, you can change the advanced default configuration in “Advanced Test Properties” section to fit your specic needs.
Console Command Line Arguments
You can optionally fill the command line arguments. For example, if you upload a file named: my_config.jmx, the command line argument should include: -t my_config.jmx. Please make sure this file is uploaded to your account using the Files tab (see instructions below).
* Please avoid using special characters like ", &,^,% in the command line arguments.
JMeterEngine Command Line Arguments
You can optionally fill this command line arguments field. These arguments are passed to each JMeterEngin in the distributed JMeter environment.
JMeter Version
PXpro can run any of the supported JMeter versions. You should use the JMeter version you are using on your local computer to create your test scripts.
Dedicated IPs
In some situations, such as a when you are connecting from behind a corporate Firewall, you would want the JMeterEngines to have dedicated IPs. We can assign dedicated IPs to the JMeterEngines. Dedicated IPs are an expensive resource. If you wish to have dedicated IPs assigned to you, please contact us.
Slow Internet Connection
Enable this check box only if you are using a low bandwidth Internet connection. In any case we highly recommend using PXpro with a strong Internet connection only. We recommend not using PXpro on a virtual desktop (for example using CITRIX). We are providing a remote application access to the remote JMeter servers. This requires a strong bandwidth Internet connection.
*** This feature is in Beta phase and can be found unstable sometimes
The Master/Slave feature synchronizes and aggregates the results from several test sessions in to a single aggregated report.
A Master/Slave setup can provide for the following:
- Aggregate the results from numerous test sessions simulating different business processes into one single report
- Aggregate the results from numerous geo locations into one aggregated single report
To use the Master/Slave setup, you need to follow these instructions:
- First, set the tests you want to run as Slave tests under the Advanced Properties field-set.
- One of the tests should be defined as Master. This test will be used to control the Slave tests.
- Once set up as Master, a list of Slave tests will appear allowing you to choose the Slave tests that will participate in the test.
That's it. Don't forget to save all the tests.
Once you run the Master test, all Slave tests will automatically run. You can view each test results at its own test panel. You can view the Master test results under Custom Report(Beta)/Master Report at the Master test panel.
You can reboot and terminate the entire session from the Master test panel. You can obviously control each unique test (only Slaves) via it's panel.
For example, we will name a test simulating business process A - Test A. We will name a test simulating business process B - Test B. We will define Test B as Slave. We will define Test A as Master. Once we run Test A (the Master), both tests will run (Master + Slave). We can view Test B report under Test B - Load Report. We can view Test A report under Test A - Load Report. We can view the aggregated report under Test A - Master Report.
*** This feature is in Beta phase and can be found unstable sometimes
JMeter user & system properties
In case you need to add properties to JMeter’s default “user.properties” file, go to the Files tab of the relevant test and upload your “user.properties” file. PXpro will identify it and will add it to the JMeterEngines and console servers. Similarly, "system.properties" can be uploaded to update system properties.
New jar files
Users that develop and/or enhance their JMeter components can upload their own JMeter components and add-ons.
To do this, jar your JMeter components and then upload the jar files. PXpro will identify the uploaded jar files and will automatically copy them to the JMETER_HOME/lib/ext directory.
System hosts file
You can control the “hosts” file of the launched servers. As a result, you are able to add additional local DNS entries as if you were editing the hosts file locally. One important use for this feature is to provide permanent DNS resolution for certain DNS names. This can reduce the number of times PXpro calls on your DNS for name resolution.
To do this, upload your “hosts.txt” file. This file will replace the "/etc/hosts" file on every launched server. Please note that the new hosts file should include at least one line of the form: 127.0.0.1 localhost. The name of the file must be "hosts.txt". The file must be uploaded prior to launching the servers. Any "hosts.txt" file uploaded during the test requires servers reboot for it to take effect.
When you run a test using a headless configuration, the test generates a JTL with the minimum JTL properties set required to generate the reports.
You can further customize the type an amount of information saved with-in the JTL file by setting the JTL Properties options in the test Edit tab.
Certain JTL properties can affect JMeter memory consumption and cause the test to break under a heavy load. Memory consuming properties are marked in red, while safe properties are marked in green. It is recommended to enable memory consuming properties only under a small capacity load or when debugging a test.
User-experience monitoring is provided by Selenium. During the load, real browsers are launched to measure end-user experience. Once launched, each browser will execute the provided Selenium script. Providing a Selenium script is optional. Instead of providing a Selenium script, you can provide several landing pages that will comprise an automatically generated Selenium script. Press the button below to see an example report generated in real time.
PXpro not only measure all render-time figures as perceived by a real browser, it also provides a complete waterfall breakdown of the entire request/response sequence. This reports enable in-depth investigation while trying to identify the cause of a certain behavior under a load scenario.
Please read our blog, How to Analyze the Results of a Large-Scale Load Test, to learn how to analyze PXpro reports.
* To generate a complete waterfall breakdown we leverage yet another excellent open-source tool: Software-is-Hard.
Automatically Generated Selenium script
First option is to provide landing pages. A script will be generated automatically simulating a user browsing through the provided landing pages.
Propriety Selenium Script
If you require a more sophisticated script, for example, a script that includes a log-in or a POST request, you can record a script using Selenium IDE and upload the propriety script to your account.
When you provide a Selenium script you need to provide a reference domain. All pages comprising the script must belong to the same domain.
This section applies only if you run in GUI mode, where you launch the JMeter GUI as opposed to running headless.
The NOMACHINE confirmation pop-up will be displayed. NOMACHINE is the remote connection application through which JMeter console is displayed on your local machine. Please check the “Always trust content from this publisher” option and click on the Run button.
If this is the first time you launch a test, the NOMACHINE applet will install a small component on your PC that will enable the remote session. This process can take up to two minutes for the first time.
Click the Continue button in order to start the remote connection session.
Wait for the wizard to finish the setup process.
Click on Yes in order to approve the connection with the remote servers in the cloud.
Wait for the JMeter console windows to open, your previously uploaded test-script is available for you to run as if you were working from your local machine.
To launch all servers related to a test, use the Launch Servers menu function. Clicking the Launch Servers link will launch a reservation of servers in the cloud. The status of this operation appears in the status area on the middle of the page. Launching a reservation can take from three to ten minutes.
The servers are launched and their status will be automatically updated. Wait to be notified that all servers are ready to run.
Once all servers are launched an applet will appear allowing access to the remote JMeter console through the NOMACHINE remote connection application.
Sometimes it takes longer for all of the servers to launch. The NOMACHINE applet will only appear after all of the server have successfuly launched. If it takes over four minutes for the entire reservation to launch, you will be presented with the option to open the JMeter console with access only to servers that did launch successfully. If you choose to follow this option, you will not have access to the servers that did not launch yet. You can always continue waiting until all servers have successfully launched.
Once all servers are launched and the applet appears, click on Continue button in order to start the remote connection session.
You can see the applet establishing the remote connection. Please wait for this process to complete.
Click on Yes in order to approve the connection with the remote servers in the cloud.
If for any reason the applet does not load, use the provided link below the applet to access the JMeter console directly using NOMACHINE client.
Once the JMeter console window is open, your previously uploaded test-script is available for you to run. Please note that you can change the script and save the changes through the remote JMeter console.
You can choose to run your load on all servers simultaneously or start each JMeterEngine separately. We recommend running no more than 300 concurrent threads for each JMeterEngine.
In order to run all servers simultaneously, click on the "Remote Start All" option under the “Run” menu.
In order to start each JMeterEngine separately, select "Remote Start" option under the “Run” menu. All available JMeterEngines are listed; click on the one you want to start.
You can also run the load on the JMeter console by clicking on the “Start” option under the “Run” menu.
Please note that we do not recommend running a load on the JMeter Console. This is because it is already consuming massive amounts of CPU capacity for calculations related to aggregating remote JMeterEngines results. Doing so may cause the JMeter Console to freeze. We do recommend to use the console to create and debug your test script prior to running any load.
The same can be done for stopping the load (by choosing “Remote Stop All”, selecting a JMeterEngine from "Remote Stop" list or clicking on “Stop” option).
PerformanceXpert provides the flexibility to change some of the test settings and restart the test. This is done as part of the test session and does not require terminating and launching the test again. This is also the best way to efficiently use your test credits when running multiple test iterations.
Restarting the test is achieved by rebooting the test session. Changing the test settings is achieved by simply editing the settings in the Edit tab.
These are only few of the use-cases possible by restarting the test:
- Moving from headless to GUI mode and back. You can debug your script using GUI mode and restart it in a headless mode and vice-versa.
- Change/add files. You can change or add files including the test configuration file (JMX file). For example, if you figured out there was something wrong with your test configuration. Please note you can not delete files during a test session.
- Change command line arguments for the console or engines.
- Restart the load.
- Pause the load. Used in conjunction with the "Delayed Start" and the "Forced Start" features, you can pause the test by restarting the test after setting it to "Delayed Start" and then use the "Forced Start" to start the test when ready. When the "Delayed Start" option is set, the test will not start until the "Forced Start" option is turned on manually.
- Change the JTL properties.
- Change the Selenium properties.
Each restart is the same as terminating the test and re-launching it again. It's much faster and you will make a more efficient use of your test credits.
It is mandatory for a reliable load test to include a testing environment that is strong enough and well configured to provide realistic traffic simulation. Any compromise with the testing environment can cause the test to appear to have excellent results providing false confidence when in fact the testing environment did not have sufficient strength to provide a realistic simulation.
PXpro provides a monitoring utility based on the open-source Cacti, that enables the user to evaluate whether the allocated resources are sufficient for the specific load test. PXpro monitors each vital sign of every load engine. Access to the monitoring system is available via the Monitoring tab during the test.
Vital signs monitoring includes:
- CPU
- Load
- Memory
- Bandwidth
If during the test, a certain resource is found close to reaching it's limit, it is recommended to increase the amount of resources allocated for the test or decrease the test intensity.
Press the button above to view a demonstration of the reports that can be generated by PXpro in real time.
To generate a report you must enable at least one JTL file.
JTL files are an inherent functionality of JMeter. You can read more about JTL files in the JMeter user guide. With this functionality you are able to have real-time reports including detailed graphs of tests results.
In order to view real-time reports, all you need to do is specify a JTL file name in one of JMeter's listeners. For example, add "Summary Report" listener and specify a JTL file name. Make sure the file ends with the extension: "jtl". Please do not change the default settings. By default the JTL format is XML and not CSV (which is good).
Once the test is running, you can view/download the JTL file itself and the resulting reports using the JTLs and/or Files tabs.
All reports are available after the test is completed. View all available reports by going to your account home page.
How to ignore Requests in the Reports
If your report includes unwanted labels such as PNGs, JPGs, CSSs etc, you can order the report not to include these labels simply by adding the word "ignore" to the label.
Any label with the word "ignore" in it will not be presented in the report.
That said, all measures of this request will appear as part of the statistics.
Available Graphs
PerformanceXpert provides three sets of reports covering every performance aspect:
- Load report include aggregated and detailed load measurements as reported by JMeter .
- Errors report details all errors that were encountered.
- User-experience report including browser render-time measurements as reported by an automated Firefox browser. This report also includes a complete waterfall breakdown of the entire requests/response sequence as perceived by a real browser during the load period.
- Page Speed optimization report according to Google's Page Speed standard. This report ranks your page speed performance aspects and also provides numerous recommendations as how to optimize your page speed to receive better search engine ranking.
Most of these reports are available only through PerformanceXpert and not through the regular JMeter interface.
Although JMeter does support few graphic reports through the regular interface it is not recommended to enable them as they consume a lot of memory and will result in the console freezing under heavy load. Use the above mentioned reports instead. They have no affect on JMeter during the test run and are accessible after the test had ended.
The "Terminate all but Console" is a special option that allows you to manage reports and results without having to pay for the servers' time other than on the Console.
By using this option you will terminate all of the load servers, but keep the GUI console open. In this case all load servers are terminated, but the load results, logs and graphs are still available for viewing and downloading.
You are only billed according to consumed computing hours. Once you terminate all load servers but the Console, you are billed only for the Console time.
To terminate only the load servers, click on Terminate menu function and choose to terminate all but the Console.
Any file generated by JMeter will appear in the Files tab and will be available to download.
Do not forget to use the Terminate menu function in order to terminate the remaining server.
If you wish to save and download the table load results displayed in JMeter GUI, click on the Save Table Data button at the bottom of the results view. In the pop-up that opens, save the results in a csv file under the default JMeter folder. The file name must NOT contain spaces.
The results file is available for download in your test page Files tab.
All test runs logs, jmx scripts, saved results and graphs files are saved and available for download during and after the load run from the Files page.
Logs, JMeter jtl files and graphs images are zipped into one file located under the Zipped folder.
It is very easy to use "CSV Data Set Config" in the cloud. Upload your CSV files using the Files tab.
In the "Filename:" field in the "CSV Data Set Config" use only the file name without any path. If you are using existing script you have created in your local machine, do not forget to erase the local path.
For further information about how to use "CSV Data Set Config" with JMeter follow this link.
If you want to use multiple JMeterEngines with a different CSV file per JMeterEngine, you will need to upload a different file for each JMeterEngines and use a parameter to differentiate each file.
Each JMeterEngine uses a distinct parameter ${__P(InstanceID)}. Please note that the "InstanceID" values start from 1 for the JMeterEngines. The value of "InstanceID" for the JMeter console is 0.
In the example above, incase you are using two JMeterEngine(s), you can upload files named file1.csv and file2.csv. In the "Filename" field you should specify: file${__P(InstanceID)}.csv.
The first JMeter engine will read file1.csv. The second will read file2.csv.
If this information is not sufficient, please contact our support.
You can use PXpro to record your script directly to the cloud environment. To do so, please follow the steps described bellow.
Launch JMeter Console
Choose Script creation only. Choosing this option will launch one server for the JMeter console and will allow you to record, write and debug your test script.
This step is not mandatory.
Discover Console Host Name
For the next step you will need a host name and port number for the proxy configuration.
The Console host name can be found Under Logs -> Network Information.
The PublicDNS field in the first line is the public host name of the JMeter Console.
Available Ports
You can choose any port from port 32000 to 64000. Any port below 32000 is closed. For example: port 32100 is a good port number, so is port 36150.
Browser Configuration
Now, configure your browser to use the JMeter Console as proxy using the host name and port number.
Further Instructions
Further instruction as how to use JMeter for recording can be found in here.
JMeter has some limitations especially when it is run in a distributed environment. Following these guidelines will assist in creating a real and continuous load:
- Do not use the Console to create your load. You can use the console for debugging purposes or to run a small load from it to make sure the script is running correctly. The GUI consumes a lot of memory under heavy load, therefore the console server by itself can not sustain a heavy load.
- Use the remote servers to create the load. Use the "Remote Start All" or "Remote Start" individual servers.
- Limit the number of threads per engine to 300. This means the total number of threads generated by your test plan should be less than 300. The overall number of threads running from PXpro will be the total number of threads multiplied by the number of JMeterEngines. For example, a test plan with 200 threads and 4 JMeterEngines, will generate a load of 800 threads.
- Disable the "View Result Tree" listener as it consumes a lot of memory and can result in the console freezing or JMeter running out of memory. It is, however, safe to use the "View Result Tree" listener with only "Errors" checked.
- Disable all JMeter graphs as they consume a lot of memory. You can view all of the real time graphs using the JTLs tab in your web interface.
- Monitor the logs. Any errors in the test plan or the test itself will appear in the log files that are available online under the Logs tab. For example: Out of Memory, disconnections etc.
- Do not forget to erase the local path from CSV Data Set Config if used.
- File names should include only alphanumeric characters, underscores or hyphens: [0-9],[aA-zZ],[_-]. File names should always include an extension.
- Clean the Files tab prior to every test run.
Best Practice Using JTL files
If you are using JTL files, we recommend following these guidelines:
- Assign a meaningful JTL file name. For example: 150210-4000t-1.jtl. Do not use spaces in the JTL file name.
- Make sure that the JTL file is saved in XML format and not CSV.
- Visit the JTLs tab at least once to generate the real-time reports.
- Use short and meaningful names for labels identifying controllers and samplers (for example HTTP requests). If you use long names, the names themselves will take the better part of the graphs generated using the JTLs tab.
