Load testing is an important step in any enterprise deployment strategy and it is a specialized skill-set that all of us need to learn.

Load testing a self-service visual analytics tool like Tableau is not easy and may even unnecessary because of the nature of self-service itself. You cannot predict self-service because people have different questions that need different solution in tableau. It is also well known that there are usually more than 1 way to solve something in Tableau.

So what is the middle-way?

As we all know, visual analytics can be split into two different use cases: data visualization & data exploration. Data visualization is analyzing information shown in dashboards and look at it from different angle by slicing it using filters. Data exploration is self-service, it may start from a dashboard or it may start from scratch without any dashboard, the latter is more common scenario.

We can do load testing for data visualization because it is always starting with dashboards. We cannot do load testing for data exploration because it is impossible to simulate a self service analytics conducted by many analysts. Don’t forget that load testing in the end is trying to simulate the actions of many users. And in the end it is still a simulation that cannot be compared with real usage from the users.

So i suggest we do load testing for data visualization only. Data exploration need a real world testing in production server with real usage from real people. Then we can use tableau server administration views to monitor the self service activities to different published data sources and tune the server performance along the way.

Ok, so what is our options to do “almost real” load testing for Tableau Server ?

=== Option 1: Scout ===

Scout is an community supported tool from tableau employee and it is good for basic validation in UAT phase and it simulate 1 single user only.

You can use Scout to answer below questions:

1. Are the dashboards loading properly without errors ?
2. Are the dashboard loading time acceptable for single user ?

So it is not suitable for real load testing with many concurrent users.

You can download scout here.

=== Option 2: TabJolt ===

Tabjolt is a community supported “point and run” scalability testing tool. It is suitable to test tableau server scalability, not dashboard response time. The response time is not accurate because various factor like think time between test and think time before interaction and it is supporting only 3 types of predetermined actions: view, interact, view & interact.

Finding tableau server “saturation pattern” is the most important job of tabjolt. Let me repeat: the most important job of tabjolt is to find how many concurrent users the server can handle before it reach saturation point and become inefficient & ineffective to process additional requests. You need to scale out and add more server to the tableau server cluster to handle the additional requests.

You can use TabJolt to answer below questions:

1. How many concurrent users tableau server can handle before saturated ?
2. If 1 server can handle 50 concurrent users, can 2 server handle 100 concurrent ?
3. Is tableau server scalable linearly ?

Tabjolt is good for scalability testing, not performance testing.

You can download Tabjolt here.

=== Option 3: Microfocus / HPE LoadRunner ===

LoadRunner is the Industry-standard Load Testing Solution.

LoadRunner is an All in One solution for performance test, load test, scalability test, error test, SLA test, with customizable real life interaction scenarios runs by many concurrent users.

They released a community edition that allow us to run the load testing scenario up to 50 concurrent users here.

IMHO Loadrunner is the best tool for load testing dashboard response time and to answer below questions:

1. What is the dashboard response time for 1 user / 10 concurrent users / 100 concurrent users with 20+ different interactions ?
2. Can we meet our dashboard response time SLA for 100 concurrent users ?

LoadRunner is like a swiss army knife that can do all sorts of load testing scenarios and we can also use it for scalability testing. With this tool you don’t need TabJolt & Scout.

Different products need different load test scenario and settings.You can do anything with this tool so what is a good testing scenario for Tableau Server ?

In this post i will share the basic of using LoadRunner and share basic scenario that i think is a good scenario to start load testing tableau server. Later i will post a more advanced testing scenario for Tableau Server and a couple of tips.

Let’s start with an overview of LoadRunner.

There are 3 important applications under LoadRunner brand that works together:

1. Virtual User Generator: generate virtual user based automated testing scripts.
2. Controller: manage & control the load test.
3. Analysis: view & analyse test results.

You start with Virtual User Generator to create various test scenarios, run it using Controller, and analyze the result with Analysis.

LoadRunner has so many bell and whistles (options) and a bit too much to create a step by step instructions. So i made instruction videos 🙂

= Virtual User Generator =

You need to published some dashboard into tableau server and if possible please use the guest account so the user no need to login. Login process may skew the final response time result. Guest account only available with tableau core license so if you have named user license then you need start from login page. I’m using guest account in this test so users don’t need to login.

For now just follow whatever i did in the videos. These steps are the result of months of trial and error so for now just follow along. I will explain the reasoning of each of my action in the next blog post.

= Controller =

Let’s continue with Controller and choose manual scenario and choose real-world schedule. Then run it and Controller will open Analysis to show you the test result.

= Analysis =

Analysis will show you the result charts. Analysis has many options but the 3 important ones are: include think time, add SLA, and export raw data. Below video will show you the charts and to include think time in the results.

Do this to add SLA to the results:

And do this to export the result raw data into Excel file for further analysis using Tableau:

That’s it for now, my future blog posts will explain the reasoning of each of the above steps and LoadRunner Tips & Tricks for Tableau Server.

Enjoy today my friends.