Many Businesses Aren’t Protecting This Valuable Asset

full frame shot of eye

A 7-Minute Introduction to DataOps

It was a typical business morning for the Technology Consulting Group in early 2000. The Internet was booming, technology was growing in every possible genre, and we were hiring. The secretary saw the new recruit’s email come up on screen. Ten minutes later, all the documents on the network had been overwritten with zeros… the document files were there, but the content was erased.

What first looked like an email containing a resumé was actually a virus. Millions of dollars of research, invoices and contacts were gone. We had backups, of course, but Iomega Jazz drives turned out to be unreliable. Within a few months, the prosperous and growing company closed its doors. All it took was losing its intellectual property – its data.

Fast-Forward to Today

There’s a new trend across many of companies today regarding data, databases and reporting. For the past few years, companies have begun treating data, its structure, and its presentation with the same regard as software managers treat code. This is because businesses have experienced some serious hurt over the past two decades by not applying DataOps principles.

  • It costs small businesses to go bankrupt. e.g. Technology Consulting Group.
  • It costs large businesses billions of dollars in damages; e.g. Samsung, Uber, Progressive.
  • It can even make large businesses go under; e.g. It nearly ended Pixar.

What is DataOps?

DataOps is a workflow process that ensures the quality, reliability, governance and security of data, schemata, queries and reports. It offers easy delivery, quick recovery, new insights, and change transparency. It captures metadata such as the person who made the change, the associate who requested the change, what exactly was changed, tickets associated with the change, and why. It amplifies feedback loops and encourages experimentation, allowing the team to learn from mistakes to achieve mastery.

DataOps takes processes from three other well known workflows in the IT industry: Agile, DevOps and LEAN. The concept has been loosely applied in software engineering businesses since the late 2000’s, but hadn’t been formalized until mid-2021.

The Trinity of DataOps

Agile methodology is a framework for project management that that focuses on broken-down iterations of work called sprints. At the start of each sprint, a set of work items is divided amongst members of the team, or placed on a kanban board. At the end of each sprint, teams reflect on the work performed to find improvements in their strategy for the next sprint.

DevOps is the practice of team-sharing files through a central repository to coordinate and collaborate within and outside the team while communicating with each other through a set of tools. DevOps provides additional tools to consolidate work, build the product, document, unit test, systems test and, if testing is successful, deploy the product.

Lean is used to continually verify the quality of a product and the security methods that protect it. For example, definitions of data can change over time, and not having a system in place to check this allows misleading data to enter the system. Data that at one time meant “A”, now means “B” and should be handled differently. Data that could compromise the company or its clients also needs to be handled in special ways to ensure privacy.

How Does DataOps Differ From DevOps?

There are several factors that are unique to DataOps, and DataOps incorporates many aspects of DevOps within its process, but DataOps is not a superset of DevOps.

FeatureDataOpsDevOps
Sharing work on the same file?Reports are siloed and SQL scripts are functionally atomic. This makes splitting work on the same entity very difficult. Pair programming is required to share development on the same file.Source code is mapped by lines that are easily split between developers. Multiple developers can check-out work on the same file at the same time.
What teams are involved?Business Operations
Data Science
Business Intelligence
Data Governance
Data Management
Data Operations
IT Operations
Compliance
Engineering
IT Operations
Software Development
Quality Assurance
Security
User Experience
Design
Operations
What skills are involved?Data management
Data science
Data analysis
Data integration
Data quality
Data security
Statistics
Reporting
Business
IT operations
Data operations
Application engineering
Data engineering
Data governance
Requirements gathering
Application architecture
Software engineering
Software development
Application integrations
Coding
Testing
Quality control
Quality assurance
Security
IT operations
Continuous Integration
Continuous Delivery
What is the pipeline like?Develop the data product
Manage the data resources (ETL)
Test to ensure quality
Release to users
Manage usage
Monitor usage and results
Design the application / changes
Develop and build the application
Test to ensure quality
Release to users
Monitor usage and error logs
Agile PlanningUsually Kanban based; some work is planned up-front of each sprint, but most work flows through the board as requests come in. Loosely structured; More organic.Usually Scrum based; all work is planned up-front of each sprint. Highly structured, mechanical and organized.
LEANFocuses on source-of-truth and data governance principles while cards are pulled or distributed from the board throughout the Sprint.Focuses on DRY, SOLID coding principles after the Sprint has begun and work has been assigned.

What tools can be used to apply DataOps?

DevOps Source Control

TFS, Subversion, or Git can hold source files like: schemas, type table seeds, utility scripts, views, functions, procedures, configuration files, and certain types of ETL packages and reports. Basically anything that you can load in notepad and read is a good candidate for Git. However, the nature of Tableau and Power BI report files requires a little more tooling.

Power BI recently added source control to its server and it is amazing! It separates the report definition from the other components and uploads the pieces up to a Git repository. You can then compare the XML of the reports’ definitions (and other components) to see what changes took place. It provides a text box for developers to comment on version changes and commit only the reports that apply.

Tableau, however, only keeps the prior 9 copies and ditches the rest; Tableau just has a rolling backup the latest 10 versions. So for Tableau reports, either Git with LFS enabled, or Bitbucket are better options than relying on the server. Either of these options allow commits with comments, versioning, tagging, merging and conflict resolution. Of the two, I would recommend a cloud-managed Git repository system.

For data changes, such as values in type or look-up tables and system data that is handled outside an administration console, such as price changes, put the data change in a script that can be checked-in, reviewed, and vetted to a test environment. Redgate provides some good tools for this such as Flyaway and SQL Data Compare.

Recommendation: Azure DevOps Git for SQL Server & Power BI, GitHub Actions for Oracle & Tableau, Redgate Flyaway and SQL Backup Pro for data

Kanban Board

Jira, Azure DevOps, Monday and Wrike all come highly recommended. Since this is the central location for filing and working off requests and features, it’s best to research which of these would be best suited for you and your team.

Recommendation: Azure DevOps for SQL Server & Power BI, Jira or Monday for Oracle & Tableau

Lean methodology

Lean methodology rests on two pillars that provide a framework for all Lean projects: Continuous improvement and respect for people. It’s more about how to use the tools, people and resources at hand to create a feedback loop that improves process and product for the client and the workers. It can use the tools already mentioned, but adapts for the unique needs of your service. Consider a system that allows you to extend it with plugins and contains workflows and pipelines to automate as much of the development and deployment as possible.

Recommendation: Azure DevOps, Jira, Monday, Jenkins, GitLab … whatever fits your team and client needs best.

Conclusion

With tools and a structured process that involves the whole team you can implement a process that protects the core components and intellectual property of your company. It allows you to efficiently and confidently release changes that effect everyone. If there’s ever a failure because of database, report, or data changes you have a way to roll back quickly. Backups can only go so far, and should be last-ditch efforts to recover from a disaster. DataOps is a methodology that keeps your data and its availability safe and operational while keeping your teams productive.

The Better Stand-Up – Part 4: The meeting purpose and meaning to its participants

In the last post, I briefly mentioned that having a purpose statement for the meeting could reduce the time it will take. That’s the focus aspect of planning, with the intent of staying on the predetermined topic and goal.

In our world, across the globe, no symbol is more ecumenically purposeful and meaningful than the bathroom sign. Everyone wants to know where it is in case they, or someone they know, needs it. That sign has a specific purpose, and is meaningful to a wide audience. This is what our meetings aspire to be. However…

There are different types of people who attend these meetings and each one will have a different goal. In a board meeting with treasurer, publicity, technical, administrative and secretarial staff present, you have each person interested in a different aspect of the meeting. Even stand-ups have a manager and sometimes a director attending along with developers and knowledge experts.

The purpose of the meeting should be distinct and focused. If everyone needs or wants to know the information that other departments provide, it should be stated what the expectations are. Perhaps the director wants to know how much a software upgrade costs and the developer wants to know if and when to expect that software to provide an estimated time of completion to the manager.

Often there are attendees who have little input and no interest in the rest of the meeting. The brief information they provide could be summed up in an email or a pre-recorded video presentation.

To make meetings more meaningful:

  1. Think of what you can provide that others cannot.
    • It doesn’t have to be unique knowledge or abilities … it might be finding some time putting slides together when other members of the team cannot spare the time for it.
  2. Consider if what you provide has interest to other attendees.
    • Put on a thinking cap and use your listening skills. Ask the other attendees if you’re unsure. Healthy teams want to improve the overall productivity and will be glad to help.
  3. Think of what you’d like from the other attendees.
    • A favor to ask
    • Information
    • Opportunity request
  4. Remember that these meetings are give and take.
    • Don’t spend too much time doing one or the other; Ideally, you want to sit on both sides. Otherwise there are better ways to achieve the goal than taking up other people’s time in meetings.

Meetings should be genuinely helpful in providing direction and cohesion to the team and its individuals. They are a critical element in the decision making process, and can be a form of team building. Making them more effective, impactful, timely, and meaningful saves money and frustration by increasing productivity and involvement.

The Better Stand-Up – Part 3: The Meeting’s Duration

Meetings are inevitable, but sometimes they become extreme. At one time, Ford had meetings to prepare for meetings.

Research published by the MIT Sloan in 2007 expressed that meetings increased in length and frequency since 1960 such that executive employees spend an average of 13 hours more a week on them.

Today, these same people spend over half of their time at work in meetings. It was determined in a 2014 Bain & Co. research that to prepare for a successful meeting, the presenter (or moderator) and his/her collective subordinates spend nearly 43x more time on collecting, organizing and preparing the content than the time spent in the meetings themselves.

“One company’s weekly senior leadership meeting directly consumed 7,000 hours per year for the attendees—but 300,000 hours company-wide among subordinates in preparation and related meetings, the study said.”

The “Stand-Up” meeting, introduced with the Scrum methodology, was created to reduce meeting times. The idea was that with everyone standing up, we would get tired on our feet and want to leave soon after the meeting started. In nearly every stand-up meeting I’ve attended in the past seven years, most, if not all, of the attendees were sitting down and they lasted longer than the 5 to 10 minute suggested time block.

From grandiose annual board meetings to mundane daily stand-ups, meetings have been found to only be effective during 1/3 of their time.

For the most part, I suggest using email, wikis, and discussion technology (like Slack or MS Teams) to disseminate information. In situations where you think the chat would go longer than 2 responses or an email would require more than 5 lines, call for a quick meeting.

We need a balance between long-threaded emails, confusing text chats and drudging group-meets. I don’t have one answer that fits the variety of daily situations, but there are some suggestions from personal experience and the experiences of others that can help.

Here are some that I found noteworthy:

Have a purpose Statement For The Meeting

I have a successful friend who refuses to attend any meeting without one. It must have a time-block, a topic and a goal like: Meeting to discuss the new client website 2-2:15, to make a decision on a WordPress theme.

Prepare for the meeting, even if you only an attendee

If you’re leading the meeting, provide something for the attendees to read or do before the meeting.

If you’re attending the meeting, treat it as a job interview with questions for yourself that you’d like answered. You don’t need to ask them in the meeting because just having them makes you more attentive and aware. Several coworkers have treated meetings this way and I’ve watched every one of them advance their career quickly. I believe there’s a correlation.

Treat each meeting differently

Not all problems are the same. Some are quick one-offs. Others are larger or more heated issues. Adjust the time in the meetings accordingly.

Use an Egg Timer

End every meeting 5 minutes early via an egg-timer. When it goes off, the meeting is over. It’s nobody’s fault – it’s just the rule. If more needs to be discussed then more time is needed to plan the discussion. Take a break and schedule the meeting at least one hour later. This gives people time to prepare.

Penalize Latecomers

Seth Godin suggests that those who come in 2-minutes later than the last person, after the meeting’s start time, contributes $10 to a collective group fund.

It’s best to work this out with the team’s culture, first. As it can ruffle some feathers telling people how to spend their money.

I had a boss/company owner who would create penalties (such as singing to the group) if they were late.

Prohibit Phones

A Harvard study shows that just having a phone, even face-down, on a table reduces attention and productivity.

Leave Unproductive Meetings Early

Seth Godin also suggests that if you’re not adding value to a meeting, leave.

I suggest you do it politely and with permission. Leaving meetings might be against the corporate culture and can come across as rude. Talk to your supervisor to gather expectations on this.