How to Streamline Testing

How to Streamline Testing

How do you maintain the quality of your software while making your testing efforts more efficient? 

Back in my first post, I lamented the way that testing and code quality seemed to be little more than lip service in my prior employment experience and how I strove to change that perspective at my current company. I am very lucky to have the opportunity to attempt these changes and I feel that there have been many successes. Yet even with that opportunity, one thing doesn’t change: there is SO MUCH that needs to be done. So how do you maintain the quality of your software while making your testing efforts more efficient? Spoiler alert: there is no silver bullet. However, here are some things that help us streamline testing.



Having a plan for testing is essential to help streamline the process. It can be as informal as a checklist or as detailed as a complete list of tests, but defining the direction and scope of testing ahead of time is a huge time saver. Beware of putting too much effort into planning. The amount of planning should match the impact of the work. There is nothing more important than time, and spending thirty minutes filling out test planning paperwork for a ten-second test kills productivity, enthusiasm, and momentum.

Pull Requests and Code Review

Testers should be looking at the changes made to code, attempting to understand those changes, and asking questions when they don’t. This will help confirm if the test plan is accurate or needs revision prior to testing. It will establish a dialogue with developers that can reveal updated or missed requirements that can lead to unnecessary delays in testing. As testers and developers build their relationships, testers improve their code understanding, which in turn improves their understanding of what and how to test.

Regression Testing First

As with planning, not every test effort will require regression testing. But when regression testing is part of the testing plan, do it first. That way, if you find that something that previously worked is broken, it’s as early in the process as you can make it. Too often, a majority of the testing time is spent testing the changes first, and then regression testing is done. This approach usually means that regression testing is short-changed. If regressions are found late in the cycle, there may not be enough time to correct them before release, leaving product owners no choice but to “accept” them to get their new feature released according to their timelines.

Multiple Testing Environments

This might seem obvious, but it’s surprising how frequently companies only have one, or even just part of one, test environment. Here’s a good return-on-investment (ROI) exercise to support more than one testing environment: If important (and potentially revenue-generating) features are delayed because they are stuck in the testing queue, do the investigating and find out how much the “delay” is “costing” the company to help make your point. Having the ability to test in parallel reduces queue time and time spent in test set up and context shifting. Even a partial second environment can have a significant impact on streamlining testing.


Yes, this is a reference to the 80s block stacking game, but the analogy is valid. This tactic is most beneficial when you have more than one test environment, but it could still apply if you have just one. Not every feature will require a complete environment for testing. This is especially true for systems that have both front-end and back-end components. Identifying those features that do not require an end-to-end test environment and deploying them simultaneously means that two (or sometimes more) features can get tested at the same time. And as above, parallelism streamlines testing.

“NOT” Testing

(Unconventional Testing)

Production Testing – Don’t shake your head like that. Read a bit first. This may not be possible for some companies, but for others, it should be seriously considered. Testing takes time and time is money. All testing has a cost that should provide a “Return on Investment”. If the risk is very low, it can be more cost-effective to roll out a change to a single production machine, a small group of production machines, or the whole system and then monitor the effects than to configure a testing environment and run tests. An example would be an ETL query that has been modified to include or exclude a type or status in the result set. New integrations and Proof of Concept (POC) work also fall into this category since the work is essentially “on spec,” with the expectation that it will provide benefits in the long run.

Over-The-Shoulder Testing – Or you could call it “Developer-Tester Peer Testing”. Test environments don’t always have the resources that production environments do, especially for data heavy jobs. When this is the case, we often do a one-on-one code demonstration, where the developer walks through the existing and updated process(es) with the tester and must provide before and after data, either to show that it hasn’t changed, or that it has changed according to the new requirements. This type of testing should be planned, and the tester should posit various use cases, scenarios or possible issues (missing data, bad data, no connection, no directory, etc.) and the developer should be able to show or demonstrate the code that addresses them. This type of testing is where developer-tester relationships are built.

Developer Testing – And before you start yelling: no, I am not advocating or in any way suggesting that developers can replace testers.  But testers should realize that good developers do test, and great developers test a lot. They write unit tests, they generate files and do differentials on the outputs, and they validate data inputs and outputs.  And if a tester doesn’t know that those tests were performed, they will test them again. This is a duplication of effort. We advocate developers documenting their unit tests and any testing they’ve done, so instead of repeating those tests, testers “validate” them.  Or they can “reject” them (after a discussion) if they feel the testing doesn’t cover what was planned. They then perform any additional planned tests (like regression tests). The objective is to reduce the duplication of testing effort.


Or more accurately, testing with tools.  All testers use tools to test. Some use more complicated tools than others.  Look for opportunities to remove manual and repetitive test operations, whether it is generating test data, validating test data, or full-blown automation suites.  If you are making your first foray into automation test suites, resist the urge to begin with your user interface, especially if it’s is evolving. You’ll end up spending more time maintaining tests than using them.  Instead, look to validate your APIs and other backend systems. The initial investment is more, but the longevity and robustness of your tests will provide a much better ROI.

Central Library

In our business, we have many customers making requests and receiving responses from us.  Most use a standard format, but quite a few need custom requests, custom responses, or both.  We maintain sample requests and expected responses in a central area. Developers, testers and even account managers use these samples to either baseline existing code before making changes, verify that code changes haven’t negatively affected expected responses or for troubleshooting.  Having a central repository saves time and reduces duplication of data. Your business may also have common data or processes shared by many groups and could benefit from a central repository.

Test Management System

This is an extension of the central repository idea that is specific to testing.  We use a test management system to house all of our tests, whether automated or manual, in one location.  Tests can be curated and grouped by descriptions and keywords to map to coverage areas. This means that regression testing can be limited to only the areas affected by the changes under test rather than running a complete regression every time.  One copy means only one place that tests need to update when features change and only one place to look when searching for test documentation with new team members.

The examples above all have one or more common threads: return on investment (ROI – or as I like to ask, “Is the juice worth the squeeze?”),  reduce or eliminate duplication of effort, and pragmatism. It also offers employees ways to embrace many of the concepts mentioned in my previous post on the Quality Mindset, including heightening awareness, extending and encouraging trust and ownership, and accepting risk.  Each company’s needs and circumstances are different, so feel free to change up what you’ve read here or to use it as a springboard to come up with your own method of streamlining your testing.

Top 4 Forbes Round-Up: Bidtellect CEO Offers Actionable Advice to Implement Now

Top 4 Forbes Round-Up: Bidtellect CEO Offers Actionable Advice to Implement Now

Bidtellect CEO Lon Otremba has offered a healthy share of advice on Forbes the last couple of weeks on everything from bad advertising to holiday campaigns. The bottom line? Stay truthful, know your audience, and test early. Here’s a round up of Lon’s October Forbes Features with actionable advice you can start implementing now.

1) Why Consumers Hate Your Brand’s Advertising

First, the bad news: consumers probably hate your advertising. Good news: Forbes tapped the experts for not one, not two, but 13 reasons why and how to fix it. But as Lon proves, sometimes good life advice is applicable to advertising advice. If you can stick to this, then all of your other tactics will fall into place, and your brand’s image will be beloved in no time.

Lon says: Not Being Truthful

I’ve found the quickest way for consumers to hate your advertising is to feed them something not genuine or truthful. Consumers can spot a phony a mile away, and when you lose credibility — as with anything in life —  it is nearly impossible to win it back. 


Read more: 13 Reasons Why Consumers May Hate Your Brand’s Advertising (And How To Fix It)

2) How Marketers Can Tell Compelling Stories 

What’s the key to all good marketing, advertising, and content marketing? Good storytelling. What makes an engaging story? So glad you asked. 

Lon says: A Compelling Story Is Stronger Than Any Pitch

The first storytelling lesson I ever learned was the most basic of all: a story is much more engaging than a pitch. So, rather than making good pitches, tell compelling and engaging stories. These are my four components of engaging stories: specific detail (not generalities); emotion; proximity (make it close and personal); having a point — a loss is as compelling as a win if there is a point to make. 


Read more: How Marketers Can Tell Compelling Stories: 13 Valuable Lessons

3) What Are the Most Important Content Marketing Metrics?

And while we’re on the topic of content marketing, Forbes’ experts agree on 13 crucial metrics to determine the quality of your content marketing strategy. But there’s one metric that will inform all your other goals.

Lon says: Audience Engagement

Tracking audience engagement is the most critical metric because it measures interaction with your brand. Developing an informed set of engagement component metrics that you can synthesize into one measurable metric is the goal.


Read more: 13 Crucial Metrics To Determine How Good A Content Marketing Program Really Is

4) How To Ramp Up Holiday Marketing in the Fall

Finally, maybe you’re reading this because you’re in the middle of/already planned/are stressing about your holiday campaigns. Yes? No? Forbes offers 12 tactics to get your holiday campaigns in tip top shape before the first echoes of holiday music pipe through speakers everywhere. According Lon, testing optimization strategies ahead of time should be top priority:

Lon says: Test Optimization Strategies Early

Use fall season to test optimization strategies. Test out different creative combinations and personalization tactics based on your key performance indicators. Consider placement level optimization, dynamic creative optimization (DCO), contextual targeting and retargeting, as well as ad types and devices. The holiday period will be your chance to reach far and wide — you don’t want to lose budget to error.


Read more: 12 Tactics To Ramp Up Your Holiday Marketing During The Fall Season

Four Common Content Marketing Mistakes to Avoid

Four Common Content Marketing Mistakes to Avoid

Today, content marketing is an essential way to build a connection. Instead of pitching products, brands are aligning their strategies to provide truly relevant and helpful content to their consumers from solving’ business problems, entertainment, or creating a lasting connection.

The modern content marketer is spending an exorbitant amount of budget, resources, and time on content creation. So why is it still not right? What’s the hold-up? These are the four common mistakes to avoid to guarantee engagement, results, and brand loyalty.

“When consumers engage with your content, they engage with your brand.”

– Lon Otremba, CEO of Bidtellect

1. Your Content is Stuck in Content Land

“Content marketing is a strategic approach focused on creating and distributing valuable, relevant, and consistent content to attract and retain a clearly-defined audience – and, ultimately, to drive profitable customer action.”

– CMI, “What is Content Marketing?”

Recently, our CEO Lon Otremba spoke at Brand Movers Edge Conference to talk about all things Content Marketing. His focus breaking down the content silos. Yes, these still exist. In fact, our VP of Sales, Terrah Bocchi wrote a great piece about it.

Let me as you a question, after you come up with a “killer content marketing idea” what is your next move? Is your first instinct to make your vision a reality, never really stopping to think about the purpose of the content or distribution? What does your marketing stack look like today? Does it look something like this?  Months after you get your “killer content marketing idea” most people are still deciding on content management platforms, content creation, and content curation companies. By then, your “killer content marketing idea” has been through so many hands and approvals it’s metamorphized into a project that’s not even your vision anymore. Oh, and it keeps you up at night.

Truth is, you probably already have valuable content to distribute. Here’s a great example: one of our clients came to us and said they were working on a great content marketing piece. They showed us the initial designs and within a day we put together Native creatives and foolproof targeting plan. A few weeks later, we reached out to the client and asked them if they were ready to distribute their content. Unfortunately, the content was stuck somewhere in creation-land and approvals. We went back to the client and said, “Well, here’s all these great articles and blog posts about your brand and they are relevant, entertaining, and witty.” The client agreed to test this idea with us. You know what? I don’t think that content marketing piece ever made the finish line, but the client has been distributing existing content with us for a few years now. They recognized the value. 

By all means, I’m not suggesting you stop creating content – we love content. But don’t get stuck in the silos. Make sure you align your distribution strategy with your creation – this is the key to your success.

– Lon Otremba, CEO of Bidtellect 

2. The Landing Page Isn’t Responsive (Or Mobile-Optimized)

“For the first time ever, US consumers will spend more time using their mobile devices than watching
TV, with smartphone use dominating that time spent. The average US adult will spend 3 hours, 43 minutes on mobile devices in 2019, just above the 3:35 spent on TV. Of time spent on mobile, US consumers will spend 2:55 on smartphones, a 9-minute increase from last year. In 2018, mobile time spent was 3:35, with TV time spent at 3:44.”

– eMarketer, “Average US Time Spent with Mobile in 2019 Has Increased

We are witnessing a radical shift in content consumption and behavior. Have you ever been on the subway, or waiting in a doctor’s office, business meeting, or even at dinner at a restaurant where everyone is not looking down at their phones? They’re swiping, liking, and consuming content. Now, more than ever, it’s important to make sure your content is responsive.

Often, especially in the case of interactive content, it’s important to ensure that your content appears correctly on the mobile page. This may require some backend work, but it’s well worth it in the end. If your content marketing objective is to get a PDF download, it makes sense to have your mobile-ready landing page email the PDF rather than download it on a mobile device (and bonus: you got another email address to remarket to). Lastly, if your end goal is a sale or transaction, remember that most people aren’t going to whip out their credit card on the subway or in a public place. Having some sort of payment transaction such as PayPal will increase your conversion rates.

We consistently see incredibly high click-through rates on mobile devices via In-Ad and In-Feed Native Advertising; however, if your landing page is not mobile-ready, engagement will decrease and bounce rates will increase. 

3. You Don’t Optimize Your Creative – Or Utilize Best Practices

Step one: Throw away all misconceptions, best practices, yadda yadda you know about traditional banner ads.

We’ve talked a lot about banner blindness. Now, step two: I want you to get display completely out of your head.

We good?

Native ad units are composed of a few core components: an image, a headline, a description and “Sponsored By”. Whereas display ads are whole, Native Ads are compiled in real-time designed to fit the form and function of the publisher’s website.

Over the last six years, our [b]+studio team has tested a plethora of strategic tactics with our Native creative – and we’ve measured performance. We know what works; we know what doesn’t work; we know what we thought wouldn’t work – does in fact work. We typically provide 3 – 5 creative variations utilizing different best practices for each landing page. (We let our optimization engine take care of the rest).

So that you don’t get started off on the wrong foot, here are a few of our Native Creative best practices.


  1. Use bright and eye-catching colors
  2. Feature people, from shoulders up in natural settings or use of the products (rather than the product in isolation.
  3. Close up photographs work better than cluttered images
  4. Trigger memories and emotions rather than neutral expressions


  1. Headlines should look like a news article
  2. Emotional, Educational, Relevant and Engaging headlines work best
  3. Facilitate concern, doubt or worry. 
  4. Offer to education or solve problems
  5. Ask Questions
  6. Listicles work! And be punny!


  1. Give the emotional “fish hook” that will real the reader in.
  2. Set up a problem and have a clear call to action as the solution
  3. Be careful not to get into click-bait territory

Avoid creative burnout! Always a creative refresh about a month in.

Shameless Plug:  Bidtellect’s in-house Native Creative Agency, [b]+studio, offers free access to help you create your custom content assets. 

4. You Forgot About Video

Research reveals that almost two-thirds of users skip pre-roll ads when possible, and three-quarters of these do so out of habit without waiting to see if the ad content is relevant or interesting (MagnaGlobal). By harnessing two thriving categories of advertising — video and native — brands can make use of a single digital super-format that connects emotionally with consumers without disrupting their experience.

Here are a few Native Video Best Practices:
  • The key to treat native video content the same way as a static native advertisement: find and use an engaging frame as a thumbnail, and be thoughtful of the creative with a text description.
  • Shorter videos (fifteen seconds or less) can significantly drive up video completions – especially on mobile.
  • Make sure the most compelling visuals and products are at the beginning of the video.
  • Make sure the video is understandable without sound.

It’s important to view your content marketing strategy holistically. The old-age saying “If yo
u build it, they will come,” doesn’t apply to content marketing. You can build a great piece of content, but without engagement – what is the purpose? By aligning your creation with distribution, results will increase dramatically. 

What You Need to Know About CCPA

It was only a matter of time before the U.S. passed its version of GDPR: The California Consumer Privacy Act, AKA CCPA (because the industry definitely needs another acronym), will go into effect January 2020.  But how similar is California’s take on privacy and will current GDPR protections comply with CCPA? 


Motivated by recent, large scale breaches of consumers’ information, including the March 2018 incident with data-mining firm Cambridge Analytica that exposed the misuse of tens of millions of people’s personal data, the CCPA’s purpose is to take greater safeguards to protect consumers’ privacy against misuse stemming from carelessness, shadiness, and outright theft and fraudulent activities. The bill grants California residents grants greater privacy and control over their data while demanding more transparency and communication from businesses.

What is it? 

Essentially, businesses must now provide explicit information on how and to whom personal data is being used, as well as honor requests for more information by consumers.  Businesses must also clearly state if they engage in selling their customer data. 

Under CCPA, businesses are required to provide California residents with the right to:

  • Know what personal data is being collected about them.
  • Know whether their personal data is sold or disclosed and to whom.
  • Say no to the sale of personal data.
  • Access their personal data.
  • Request a business to delete any personal information about a consumer collected from that consumer.
  • Not be discriminated against for exercising their privacy rights.

As such, businesses are required to notify and request permission from customers before collecting data, state its purpose, use the data in a lawful manner, and comply with consumers’ requests for deletion.

When does it go into effect?

January 1, 2020 (confusing as it is called “The California Consumer Privacy Act of 2018.”). It becomes enforceable on July 1, 2020. 

Who does it apply to?

The CCPA applies to any for-profit businesses (not nonprofits or governmental entities) in the state of California that collect consumers’ personal data and meets at least one of the following criteria:

  • Has annual gross revenues in excess of $25 million
  • Handles data of more than 50,000 people or devices
  • Earns more than 50% of its annual revenue from selling consumers’ personal information.

Is it really California only?

California is an important state to set a privacy precedent. Not only does it hold the largest population in the United States (39.56 million in 2018), but it’s home to the hot bed incubator of tech powerhouses and cutting edge startups. Notable digital companies headquartered in California include Alphabet/Google, Apple, Facebook, and Oracle.

While the law only applies to customers that live in California, most companies will have to shift privacy policies to accommodate it. Other states will likely follow suit and use the CCPA as an example to set their own state-level privacy laws. 


The good news is that CCPA and the European General Data Protection Regulation (GDPR) have many similarities, so companies that have adopted practices that comply with GDPR will be pretty well-prepared for CCPA. Both the CCPA and the GDPR adopt an expansive definition of personally identifiable information (PII) and value the customer’s right to choose and understand how their data is being used.  There are some differences – overall, CCPA is more specific in their requirements, while GDPR is a bit broader. 

PWC offers a great table comparing the two on main points, from scope to enforcement:

What about child data: CCPA vs. GDPR vs. COPPA?

The protection of child data is not new in the US: the Children’s Online Privacy Protection Act (COPPA) came into effect in 1998. (In Europe, child data was treated like every other piece of personal data until the GDPR set specific and stronger rules.) Now, the CCPA goes even further than COPPA in children’s data protection: While all consumers can opt-out of the sharing of their information, consumers under the age of 16 must opt-in. And if they’re under 13, their parents or guardians must opt-in (EdSurge, 2018).

What do consumers think?

67% of US online adults and 57% of European (EU-5) online adults are not comfortable with companies sharing and selling their data and online activities, according to Forrester research. And 51% of US online adults and 48% of EU-5 online adults report taking active measures to limit the collection of their data by apps and websites (“Tackle The California Consumer Privacy Act Now” Forrester Research, Inc., February 8, 2019).

55% of US privacy professionals plan to be CCPA-compliant prior to January 1, 2020. 25% plan to be ready for July 1, 2020, when the law becomes enforceable (International Association of Privacy Professionals (IAPP) and OneTrust via eMarketer).



To Have A Quality Mindset, You Need A Quality Environment

To Have A Quality Mindset, You Need A Quality Environment

In Part 2 of the Quality Series, why a Quality Mindset requires a Quality Environment…And how to build it.

Karl Hentschel

“Quality is never an accident. It is always the result of intelligent effort.”

– John Ruskin (1819-1900)

“Quality is not a consequence of following some set of behaviors. Rather, it is a prerequisite and a mindset you must have BEFORE you decide what you are setting out to do.”

– Edwin Catmull (1945- )

In my previous post, I explained how Bidtellect envisioned changing “Quality Assurance” from a step in the software development process into a shared Quality Mindset. But a Quality Mindset requires a Quality Environment in which to work.  The quotes above led me to many books, two of which, Good to Great by James C. Collins and Creativity, Inc. by Edwin E. Catmul, were instrumental in helping define a quality working environment.

The concepts below are intended for everyone in a company, but are especially intended for company leaders.  Note that I didn’t say “manager” or “coordinator” or “director” (although it really helps if leaders with those titles buy-in to the quality work environment concept).  Company leaders are the people in your organization to whom others look because they get the work done.

1. Develop and Support Competence


This may seem like a no-brainer, but in order for people to have a Quality Mindset, they need to know what they’re doing.  That means, they either bring experience to the company, the company provides experience to the person, or a combination of the two.  Don’t ask a plumber to build a brick wall or a mason to wire the lights, unless you plan on training them properly.

2. Model Discipline

But not the kind you’re probably thinking of right now.  I’m talking about internal discipline, not external discipline.  Not a manager, a director or vice president enforcing rules, but rather individuals doing the work to the best of their ability.  (Notice I said “work”, not “job”. This is important for later.) Discipline is (and should be) a modeled behavior and takes the form of disciplined thought and disciplined action.

3. Extend and Build Trust

Disciplined people, engaging in disciplined thought and taking disciplined actions don’t need to be managed.  They will do what needs to be done without needing to be told, if you trust them to do it.

4. Extend and Encourage Ownership

Ownership and trust or kind of “chicken-and-egg” because each fosters and reinforces the other.  I have a standing rule for my Quality team: You don’t have to ask permission to take responsibility.  Responsibility doesn’t necessarily mean you will do the work, but it is expected that if you take ownership, you are responsible for ensuring it gets done.  This ownership isn’t limited by your “job” but rather the work that needs to be done.

5. Heighten Understanding

A leader is responsible for heightening people’s awareness of what they do not know.  Do not confuse this with competence (although for junior people, it may by applicable).  This is more about making people aware of what occurs outside of their competencies. Get people thinking about “the company” as a whole, rather than just the testers or the developers or the technology team, etc.

6. Show (Frequent) Appreciation

Appreciation is often regarded as a leadership responsibility and it most certainly is.  But that doesn’t mean leaders are the only ones who should recognize individual or group accomplishments or appreciation for hard work, diligence or tenacity.  They should model appreciation and encourage it amongst team members.

7. Clear Obstacles

Bureaucracy is a reaction to incompetence and lack of discipline.  It is a leader’s responsibility to get the right people, facilitate the right chemistry, and develop and support them.  Once they engage in discipline and take ownership, the controls can be loosened and the bureaucracy decreased or removed.

8. Accept Risk

… and with it the mess it creates.  If you trust your people and allow them ownership, there will be times when they will get it wrong.  You should NOT be looking for ways to keep people from making mistakes. A true leader will enable people to resolve problems WITHOUT BLAME.  This reinforces the trust and ownership (above) and KEEPS FEAR AT BAY. People who are afraid they will get blamed, afraid they will become unemployed or afraid of ignominy DO NOT trust or take ownership…nor do they create their BEST work in the end.

Read More on the Bidtellectual Blog

Follow Us