OT? How to 'politely' point out data integrity problems?

Hi all.

Given a database that comprises a number of data batches, each
batch consisting of several different data sets - e.g. as logged by
several instuments - from the same measurement.

Each individual data set is logged in the inventory, and the theory
is that the inventory logs from the individual data sets should match
the inventory entry from the data batches.

But they don't.

The inventory logs from the data sets don't match each other, and
they don't match the data batch inventories. The database is, to
some extent, useless because of this, since no one knows what
is really in there, and what is missing.

However, if one examins the actual contents of the various data sets
one can deduce where the individual measurements were made. When
I do examine the data contents, I find that the contents match across
the board with more than enough significant digits that I trust the
results.

So now I have a problem:

1) I know - and can demonstrate - that the inventory is a mess
2) I know - and can demonstrate - how to correct the situation
3) All I need to do is to tell somebody that their inventory
   system is a mess, that anyone that examines the data set
   in more than half a glance will realize the fact, and that the
   cause is poor attention to detail in the specification.

I don't understand much human psychology; I don't understand
how or why people might set up such information handling systems
*without* also ensuring data integrity wherever possible.

So how would one address this?

Based on previous experiences I would expect the people in
charge to either snap to 'ostrage' mode, sticking their heads
in the sand: "This is how we always did, and it was never a
problem." Or they will attack me personally for either being a
troubleraiser or accusing them of incompetence.

Again, based on previous experiece, the problem is not the
way or manner my views are presented, but the contents:
It is fairly obvious to anyone that a systematic mis-match
between inventories and contents is a big issue, and one that
the people in charge ought to have both recognized and handled
a long time ago.

Any suggestions on how to *both* present such views *and*
avoid the fallout when the s**t hits the fan?

Rune

Rune Allnor wrote:
> Hi all.
> 
> Given a database that comprises a number of data batches, each
> batch consisting of several different data sets - e.g. as logged by
> several instuments - from the same measurement.
> 
> Each individual data set is logged in the inventory, and the theory
> is that the inventory logs from the individual data sets should match
> the inventory entry from the data batches.
> 
> But they don't.
> 
> The inventory logs from the data sets don't match each other, and
> they don't match the data batch inventories. The database is, to
> some extent, useless because of this, since no one knows what
> is really in there, and what is missing.
> 
> However, if one examines the actual contents of the various data sets
> one can deduce where the individual measurements were made. When
> I do examine the data contents, I find that the contents match across
> the board with more than enough significant digits that I trust the
> results.
> 
> So now I have a problem:
> 
> 1) I know - and can demonstrate - that the inventory is a mess
> 2) I know - and can demonstrate - how to correct the situation
> 3) All I need to do is to tell somebody that their inventory
>    system is a mess, that anyone that examines the data set
>    in more than half a glance will realize the fact, and that the
>    cause is poor attention to detail in the specification.
> 
> I don't understand much human psychology; I don't understand
> how or why people might set up such information handling systems
> *without* also ensuring data integrity wherever possible.
> 
> So how would one address this?
> 
> Based on previous experiences I would expect the people in
> charge to either snap to 'ostrage' mode, sticking their heads
> in the sand: "This is how we always did, and it was never a
> problem." Or they will attack me personally for either being a
> troubleraiser or accusing them of incompetence.
> 
> Again, based on previous experiece, the problem is not the
> way or manner my views are presented, but the contents:
> It is fairly obvious to anyone that a systematic mis-match
> between inventories and contents is a big issue, and one that
> the people in charge ought to have both recognized and handled
> a long time ago.
> 
> Any suggestions on how to *both* present such views *and*
> avoid the fallout when the s**t hits the fan?

Don't put it in terms of correcting a (obvious, you dumb shit heads!) 
mistake, but rather in terms of making the data even more convenient to 
use. Instead of "This is wrong", "That would be better". And hold your 
breath. And bite your tongue.

Jerry
-- 
Engineering is the art of making what you want from things you can get.
�����������������������������������������������������������������������

On 25 Feb, 11:12, Jerry Avins <j...@ieee.org> wrote:
> Rune Allnor wrote:
> > Hi all.
>
> > Given a database that comprises a number of data batches, each
> > batch consisting of several different data sets - e.g. as logged by
> > several instuments - from the same measurement.
>
> > Each individual data set is logged in the inventory, and the theory
> > is that the inventory logs from the individual data sets should match
> > the inventory entry from the data batches.
>
> > But they don't.
>
> > The inventory logs from the data sets don't match each other, and
> > they don't match the data batch inventories. The database is, to
> > some extent, useless because of this, since no one knows what
> > is really in there, and what is missing.
>
> > However, if one examines the actual contents of the various data sets
> > one can deduce where the individual measurements were made. When
> > I do examine the data contents, I find that the contents match across
> > the board with more than enough significant digits that I trust the
> > results.
>
> > So now I have a problem:
>
> > 1) I know - and can demonstrate - that the inventory is a mess
> > 2) I know - and can demonstrate - how to correct the situation
> > 3) All I need to do is to tell somebody that their inventory
> > =A0 =A0system is a mess, that anyone that examines the data set
> > =A0 =A0in more than half a glance will realize the fact, and that the
> > =A0 =A0cause is poor attention to detail in the specification.
>
> > I don't understand much human psychology; I don't understand
> > how or why people might set up such information handling systems
> > *without* also ensuring data integrity wherever possible.
>
> > So how would one address this?
>
> > Based on previous experiences I would expect the people in
> > charge to either snap to 'ostrage' mode, sticking their heads
> > in the sand: "This is how we always did, and it was never a
> > problem." Or they will attack me personally for either being a
> > troubleraiser or accusing them of incompetence.
>
> > Again, based on previous experiece, the problem is not the
> > way or manner my views are presented, but the contents:
> > It is fairly obvious to anyone that a systematic mis-match
> > between inventories and contents is a big issue, and one that
> > the people in charge ought to have both recognized and handled
> > a long time ago.
>
> > Any suggestions on how to *both* present such views *and*
> > avoid the fallout when the s**t hits the fan?
>
> Don't put it in terms of correcting a (obvious, you dumb shit heads!)
> mistake, but rather in terms of making the data even more convenient to
> use. Instead of "This is wrong", "That would be better". And hold your
> breath. And bite your tongue.

I have too much faith in Homo Sapiens - I am just unable to
comprehend the possibility that somebody in charge, who sees
these kinds of corrections, will *not* infer any corrollaries
wrt their own competence.  Which may or may not be warranted,
depending on what competence they are expected or supposed to
have in the first place.

Rune

On Feb 25, 5:53=A0am, Rune Allnor <all...@tele.ntnu.no> wrote:
> On 25 Feb, 11:12, Jerry Avins <j...@ieee.org> wrote:
>
>
>
> > Rune Allnor wrote:
> > > Hi all.
>
> > > Given a database that comprises a number of data batches, each
> > > batch consisting of several different data sets - e.g. as logged by
> > > several instuments - from the same measurement.
>
> > > Each individual data set is logged in the inventory, and the theory
> > > is that the inventory logs from the individual data sets should match
> > > the inventory entry from the data batches.
>
> > > But they don't.
>
> > > The inventory logs from the data sets don't match each other, and
> > > they don't match the data batch inventories. The database is, to
> > > some extent, useless because of this, since no one knows what
> > > is really in there, and what is missing.
>
> > > However, if one examines the actual contents of the various data sets
> > > one can deduce where the individual measurements were made. When
> > > I do examine the data contents, I find that the contents match across
> > > the board with more than enough significant digits that I trust the
> > > results.
>
> > > So now I have a problem:
>
> > > 1) I know - and can demonstrate - that the inventory is a mess
> > > 2) I know - and can demonstrate - how to correct the situation
> > > 3) All I need to do is to tell somebody that their inventory
> > > =A0 =A0system is a mess, that anyone that examines the data set
> > > =A0 =A0in more than half a glance will realize the fact, and that the
> > > =A0 =A0cause is poor attention to detail in the specification.
>
> > > I don't understand much human psychology; I don't understand
> > > how or why people might set up such information handling systems
> > > *without* also ensuring data integrity wherever possible.
>
> > > So how would one address this?
>
> > > Based on previous experiences I would expect the people in
> > > charge to either snap to 'ostrage' mode, sticking their heads
> > > in the sand: "This is how we always did, and it was never a
> > > problem." Or they will attack me personally for either being a
> > > troubleraiser or accusing them of incompetence.
>
> > > Again, based on previous experiece, the problem is not the
> > > way or manner my views are presented, but the contents:
> > > It is fairly obvious to anyone that a systematic mis-match
> > > between inventories and contents is a big issue, and one that
> > > the people in charge ought to have both recognized and handled
> > > a long time ago.
>
> > > Any suggestions on how to *both* present such views *and*
> > > avoid the fallout when the s**t hits the fan?
>
> > Don't put it in terms of correcting a (obvious, you dumb shit heads!)
> > mistake, but rather in terms of making the data even more convenient to
> > use. Instead of "This is wrong", "That would be better". And hold your
> > breath. And bite your tongue.
>
> I have too much faith in Homo Sapiens - I am just unable to
> comprehend the possibility that somebody in charge, who sees
> these kinds of corrections, will *not* infer any corrollaries
> wrt their own competence. =A0Which may or may not be warranted,
> depending on what competence they are expected or supposed to
> have in the first place.
>
> Rune

Instead of telling them anything, why don't you ask them about it?
Surely you are not the only person to notice a problem.  Why not ask
for some help with some anomalies you have spotted and see if they
figure out that the problem is not isolated to a few points, but
rather is widespread?

If you remember the Space Shuttle Challenger disaster, one of the
contributing factors was that the engineers, who had data showing that
the solid rocket O-rings would fail at the temperatures they were
launching in, were not able to effectively communicate that to
management.  You need to find an appropriate way to communicate the
facts without causing fud, fear, uncertainty and doubt.

I guess I'm not clear on why this would be hard to do.  If you present
the facts, without interpretation, then guide them through the process
of understanding the implications on their own, I don't see how they
could reach any conclusion other than that they have a problem that
needs to be addressed.

Rick

Rick

On 2/25/2010 1:39 AM, Rune Allnor wrote:
> Hi all.
>
> Given a database that comprises a number of data batches, each
> batch consisting of several different data sets - e.g. as logged by
> several instuments - from the same measurement.
>
> Each individual data set is logged in the inventory, and the theory
> is that the inventory logs from the individual data sets should match
> the inventory entry from the data batches.
>
> But they don't.
>
> The inventory logs from the data sets don't match each other, and
> they don't match the data batch inventories. The database is, to
> some extent, useless because of this, since no one knows what
> is really in there, and what is missing.
>
> However, if one examins the actual contents of the various data sets
> one can deduce where the individual measurements were made. When
> I do examine the data contents, I find that the contents match across
> the board with more than enough significant digits that I trust the
> results.
>
> So now I have a problem:
>
> 1) I know - and can demonstrate - that the inventory is a mess
> 2) I know - and can demonstrate - how to correct the situation
> 3) All I need to do is to tell somebody that their inventory
>     system is a mess, that anyone that examines the data set
>     in more than half a glance will realize the fact, and that the
>     cause is poor attention to detail in the specification.
>
> I don't understand much human psychology; I don't understand
> how or why people might set up such information handling systems
> *without* also ensuring data integrity wherever possible.
>
> So how would one address this?
>
> Based on previous experiences I would expect the people in
> charge to either snap to 'ostrage' mode, sticking their heads
> in the sand: "This is how we always did, and it was never a
> problem." Or they will attack me personally for either being a
> troubleraiser or accusing them of incompetence.
>
> Again, based on previous experiece, the problem is not the
> way or manner my views are presented, but the contents:
> It is fairly obvious to anyone that a systematic mis-match
> between inventories and contents is a big issue, and one that
> the people in charge ought to have both recognized and handled
> a long time ago.
>
> Any suggestions on how to *both* present such views *and*
> avoid the fallout when the s**t hits the fan?
>
> Rune

You might ask those guys at the Climatic Research Unit in the UK.   They 
seem to have some experience dealing with faulty databases.

;)

If you can objectively show how the database is broken, with some 
concrete examples, that should help go a long way.   As has been 
suggested, you can always pose that as a question, i.e., "This is 
supposed to match this, but these examples seem to not match.  Am I 
missing something or is this broken?"   Or something like that. 
Knowing who to pose the question to for best results is sometimes key.

Having a solution in hand, especially if it helps to verify the problem 
in the first place, should help the case.

-- 
Eric Jacobsen
Minister of Algorithms
Abineau Communications
http://www.abineau.com

Rune Allnor wrote:

> Hi all.
> 
> Given a database that comprises a number of data batches, each
> batch consisting of several different data sets - e.g. as logged by
> several instuments - from the same measurement.
> 
> Each individual data set is logged in the inventory, and the theory
> is that the inventory logs from the individual data sets should match
> the inventory entry from the data batches.
> 
> But they don't.
> So now I have a problem:
> 
> 1) I know - and can demonstrate - that the inventory is a mess
> 2) I know - and can demonstrate - how to correct the situation
> 3) All I need to do is to tell somebody that their inventory
>    system is a mess, that anyone that examines the data set
>    in more than half a glance will realize the fact, and that the
>    cause is poor attention to detail in the specification.
> 
> I don't understand much human psychology; I don't understand
> how or why people might set up such information handling systems
> *without* also ensuring data integrity wherever possible.

Dr. Rune

What is your goal exactly?

1) Satisfy your feeling of self importance?
2) Beat some idiot into the ground?
3) Exercise muscles by solving a problem?
4) Gain some money?
5) Gain political advantage?
6) Express your disagreement and dissatisfaction in general?
7) Anything else?

Answer this question honestly for yourself; there are no right or wrong 
answers. Then proceed accordingly.

VLV

On Feb 25, 2:53=A0am, Rune Allnor <all...@tele.ntnu.no> wrote:
> On 25 Feb, 11:12, Jerry Avins <j...@ieee.org> wrote:
>
>
>
> > Rune Allnor wrote:
> > > Hi all.
>
> > > Given a database that comprises a number of data batches, each
> > > batch consisting of several different data sets - e.g. as logged by
> > > several instuments - from the same measurement.
>
> > > Each individual data set is logged in the inventory, and the theory
> > > is that the inventory logs from the individual data sets should match
> > > the inventory entry from the data batches.
>
> > > But they don't.
>
> > > The inventory logs from the data sets don't match each other, and
> > > they don't match the data batch inventories. The database is, to
> > > some extent, useless because of this, since no one knows what
> > > is really in there, and what is missing.
>
> > > However, if one examines the actual contents of the various data sets
> > > one can deduce where the individual measurements were made. When
> > > I do examine the data contents, I find that the contents match across
> > > the board with more than enough significant digits that I trust the
> > > results.
>
> > > So now I have a problem:
>
> > > 1) I know - and can demonstrate - that the inventory is a mess
> > > 2) I know - and can demonstrate - how to correct the situation
> > > 3) All I need to do is to tell somebody that their inventory
> > > =A0 =A0system is a mess, that anyone that examines the data set
> > > =A0 =A0in more than half a glance will realize the fact, and that the
> > > =A0 =A0cause is poor attention to detail in the specification.
>
> > > I don't understand much human psychology; I don't understand
> > > how or why people might set up such information handling systems
> > > *without* also ensuring data integrity wherever possible.
>
> > > So how would one address this?
>
> > > Based on previous experiences I would expect the people in
> > > charge to either snap to 'ostrage' mode, sticking their heads
> > > in the sand: "This is how we always did, and it was never a
> > > problem." Or they will attack me personally for either being a
> > > troubleraiser or accusing them of incompetence.
>
> > > Again, based on previous experiece, the problem is not the
> > > way or manner my views are presented, but the contents:
> > > It is fairly obvious to anyone that a systematic mis-match
> > > between inventories and contents is a big issue, and one that
> > > the people in charge ought to have both recognized and handled
> > > a long time ago.
>
> > > Any suggestions on how to *both* present such views *and*
> > > avoid the fallout when the s**t hits the fan?
>
> > Don't put it in terms of correcting a (obvious, you dumb shit heads!)
> > mistake, but rather in terms of making the data even more convenient to
> > use. Instead of "This is wrong", "That would be better". And hold your
> > breath. And bite your tongue.
>
> I have too much faith in Homo Sapiens - I am just unable to
> comprehend the possibility that somebody in charge, who sees
> these kinds of corrections, will *not* infer any corrollaries
> wrt their own competence. =A0Which may or may not be warranted,
> depending on what competence they are expected or supposed to
> have in the first place.
>
> Rune

As one of Robert Heinlien's characters said:

"Never appeal to a man's "better nature." He may not have one. ..."

Appeal instead to his self interest.

If a manager has had employees who have, as you, correctly interpreted
the data and been able to use it he or she may not see a problem that
requires the expense of a fix. If you want someone to buy insurance,
sell the risk.

Dale B. Dalrymple

On 25 Feb, 17:08, Vladimir Vassilevsky <nos...@nowhere.com> wrote:
> Rune Allnor wrote:
> > Hi all.
>
> > Given a database that comprises a number of data batches, each
> > batch consisting of several different data sets - e.g. as logged by
> > several instuments - from the same measurement.
>
> > Each individual data set is logged in the inventory, and the theory
> > is that the inventory logs from the individual data sets should match
> > the inventory entry from the data batches.
>
> > But they don't.
> > So now I have a problem:
>
> > 1) I know - and can demonstrate - that the inventory is a mess
> > 2) I know - and can demonstrate - how to correct the situation
> > 3) All I need to do is to tell somebody that their inventory
> > =A0 =A0system is a mess, that anyone that examines the data set
> > =A0 =A0in more than half a glance will realize the fact, and that the
> > =A0 =A0cause is poor attention to detail in the specification.
>
> > I don't understand much human psychology; I don't understand
> > how or why people might set up such information handling systems
> > *without* also ensuring data integrity wherever possible.
>
> Dr. Rune
>
> What is your goal exactly?

To get some real work done. The database in question
pretty much sets the stage for what I'd like to do.
If not the database itself, so at least the spec needs
to be 100% up to scratch for my little tricks to work.

My being able to get a 100% accurate inventory will make
no impact anywhere if the spec never states that the
inventory is expected to be 100% accurate. And of course,
the stuff I am *really* interested in doing requires the
inventory to be 100% accurate.

Rune

Rune Allnor wrote:
> Hi all.
> 
> Given a database that comprises a number of data batches, each
> batch consisting of several different data sets - e.g. as logged by
> several instuments - from the same measurement.
> 
> Each individual data set is logged in the inventory, and the theory
> is that the inventory logs from the individual data sets should match
> the inventory entry from the data batches.
> 
> But they don't.
> 
> The inventory logs from the data sets don't match each other, and
> they don't match the data batch inventories. The database is, to
> some extent, useless because of this, since no one knows what
> is really in there, and what is missing.
> 
> However, if one examins the actual contents of the various data sets
> one can deduce where the individual measurements were made. When
> I do examine the data contents, I find that the contents match across
> the board with more than enough significant digits that I trust the
> results.
> 
> So now I have a problem:
> 
> 1) I know - and can demonstrate - that the inventory is a mess
> 2) I know - and can demonstrate - how to correct the situation
> 3) All I need to do is to tell somebody that their inventory
>    system is a mess, that anyone that examines the data set
>    in more than half a glance will realize the fact, and that the
>    cause is poor attention to detail in the specification.
> 
> I don't understand much human psychology; I don't understand
> how or why people might set up such information handling systems
> *without* also ensuring data integrity wherever possible.
> 
> So how would one address this?
> 
> Based on previous experiences I would expect the people in
> charge to either snap to 'ostrage' mode, sticking their heads
> in the sand: "This is how we always did, and it was never a
> problem." Or they will attack me personally for either being a
> troubleraiser or accusing them of incompetence.
> 
> Again, based on previous experiece, the problem is not the
> way or manner my views are presented, but the contents:
> It is fairly obvious to anyone that a systematic mis-match
> between inventories and contents is a big issue, and one that
> the people in charge ought to have both recognized and handled
> a long time ago.
> 
> Any suggestions on how to *both* present such views *and*
> avoid the fallout when the s**t hits the fan?
> 
> Rune

"Gawd this inventory is all @#$%ed up!  Whoever is in charge is a 
complete idiot!"

A genius will fire you then quietly fix things.  A saint will fix things 
and not fire you.  Normal humans will throw you overboard (if you're at 
sea), then wait until you're being attacked by sharks to 'notice' that 
you've 'accidentally fallen', then ignore the problem.

"I notice that there are some inexplicable problems with the inventory 
system -- look at how the index doesn't correlate with the data".

This is much less threatening, but still requires someone to admit to 
themselves that Something Might Be Wrong, and go look.  You may be 
ignored, but at least your feet will be on steel decking instead of in 
deep cold water.

"I'm just an egghead consultant, I'm a genius with algorithms, but I 
just can't wrap my head around how to correlate the index of this 
database with the data itself.  If you'll help me figure it out I'll buy 
you a beer."

This gets them to go look, and lets them be the Hero who Discovered the 
Problem.  It doesn't get you much credit unless they (a) realize that 
they've been manipulated into stumbling into the problem and (b) 
appreciate what you've done for them.  But if getting the problem solved 
means more to you than getting hailed as the Savior of the Database then 
it's an approach.

--
Tim Wescott
Control system and signal processing consulting
www.wescottdesign.com

On Feb 25, 11:44=A0am, Rune Allnor <all...@tele.ntnu.no> wrote:
> On 25 Feb, 17:08, Vladimir Vassilevsky <nos...@nowhere.com> wrote:
>
>
>
> > Rune Allnor wrote:
> > > Hi all.
>
> > > Given a database that comprises a number of data batches, each
> > > batch consisting of several different data sets - e.g. as logged by
> > > several instuments - from the same measurement.
>
> > > Each individual data set is logged in the inventory, and the theory
> > > is that the inventory logs from the individual data sets should match
> > > the inventory entry from the data batches.
>
> > > But they don't.
> > > So now I have a problem:
>
> > > 1) I know - and can demonstrate - that the inventory is a mess
> > > 2) I know - and can demonstrate - how to correct the situation
> > > 3) All I need to do is to tell somebody that their inventory
> > > =A0 =A0system is a mess, that anyone that examines the data set
> > > =A0 =A0in more than half a glance will realize the fact, and that the
> > > =A0 =A0cause is poor attention to detail in the specification.
>
> > > I don't understand much human psychology; I don't understand
> > > how or why people might set up such information handling systems
> > > *without* also ensuring data integrity wherever possible.
>
> > Dr. Rune
>
> > What is your goal exactly?
>
> To get some real work done. The database in question
> pretty much sets the stage for what I'd like to do.
> If not the database itself, so at least the spec needs
> to be 100% up to scratch for my little tricks to work.
>
> My being able to get a 100% accurate inventory will make
> no impact anywhere if the spec never states that the
> inventory is expected to be 100% accurate. And of course,
> the stuff I am *really* interested in doing requires the
> inventory to be 100% accurate.
>
> Rune

Yeah, politics can be a PITA.  Mostly it comes from people being too
stupid to make good decisions, or too intimidated or too any number of
other things.  I have had managers tell me that they understand they
are making a seriously stupid decision (not in those words *exactly*)
but that it was the decision made and we were to live with it.  The
impact was to have a single, high risk, approach to a problem when we
could have pursued more than one in parallel and used the one that
worked best.  I explained that the failure of this one item would at
least delay the project by months or worse, cause its failure.  The
guy I was talking to was military reserve and didn't care about the
facts.  In the end the vendor backed out on us (just before the dot
com bubble burst and they had bigger fish to fry) and we were delayed
by months switching to another approach.  But the guy had already
gotten his promotion so he didn't care.  I can only assume that he had
some personal reason for not reducing the risk on this project.

That wasn't the only bone head decision he made and I never understood
his reasons.  In the end I went into consulting and they hired me as
my first gig!  I got a lot more independence then, but still had to
deal with the same bonehead.

Rick