[Wikimetrics] Asynchronous Cohort Upload Deployed

[Dan Andreescu]

Dear Wikimetrics users,

I've just deployed asynchronous cohort upload. This is feature #818:
https://mingle.corp.wikimedia.org/projects/analytics/cards/818 and
basically allows you to upload larger cohorts because validation is
happening behind the scenes. I'll go over how the new functionality works
here, and will rely on one of you to point me to the appropriate on-wiki
place to update documentation.

So basically, visiting /cohorts and clicking "Upload Cohort" works as
before. But once you click "Upload CSV", your form is validated,
processed, and you're taken back to the cohorts page. Your new cohort is
immediately created but is not yet validated. While it validates, you'll
see the validation status and have a few options:

* Remove Cohort. This is destructive and will remove this cohort from your
list. Use this in case you made a mistake, uploaded the wrong file, etc.
* Validate Again. This will run validation again. One possible use for it
is, let's say you upload a cohort with some *very* newly registered users.
And because of replication lag to the labsdb databases, most of them come
up invalid. You can then run validation again.
* Refresh. This just refreshes the status of the validation and will
update the counts that show up below.

You will not have the "Create Report" option until validation is done. And
when you do create a report, only valid users will be considered and used
in the output.

One caveat. Validation is still slow. And the time limit for the
asynchronous task is set to 1 hour. I have some ideas for making this
faster by batching, and I can increase the time limit per task (but that
has other repercussions). For now, just keep in mind that the theoretical
maximum cohort size you should upload is roughly 18,000 users. I would
love some feedback about whether it's ok to increase the time limit or if
people want me to focus on making validation faster.

Dan
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://lists.wikimedia.org/pipermail/wikimetrics/attachments/20131121/2bb38fbf/attachment.html>

[Dario Taraborelli]

thanks Dan, this is awesome – I’ll give it a try this morning with some of the recent mobile cohorts.

Post by Dan Andreescu
Dear Wikimetrics users,
I've just deployed asynchronous cohort upload. This is feature #818: https://mingle.corp.wikimedia.org/projects/analytics/cards/818 and basically allows you to upload larger cohorts because validation is happening behind the scenes. I'll go over how the new functionality works here, and will rely on one of you to point me to the appropriate on-wiki place to update documentation.
* Remove Cohort. This is destructive and will remove this cohort from your list. Use this in case you made a mistake, uploaded the wrong file, etc.
* Validate Again. This will run validation again. One possible use for it is, let's say you upload a cohort with some *very* newly registered users. And because of replication lag to the labsdb databases, most of them come up invalid. You can then run validation again.
* Refresh. This just refreshes the status of the validation and will update the counts that show up below.
You will not have the "Create Report" option until validation is done. And when you do create a report, only valid users will be considered and used in the output.
One caveat. Validation is still slow. And the time limit for the asynchronous task is set to 1 hour. I have some ideas for making this faster by batching, and I can increase the time limit per task (but that has other repercussions). For now, just keep in mind that the theoretical maximum cohort size you should upload is roughly 18,000 users. I would love some feedback about whether it's ok to increase the time limit or if people want me to focus on making validation faster.
Dan
_______________________________________________
Wikimetrics mailing list
Wikimetrics at lists.wikimedia.org
https://lists.wikimedia.org/mailman/listinfo/wikimetrics

-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://lists.wikimedia.org/pipermail/wikimetrics/attachments/20131121/34e17347/attachment.html>

[Dario Taraborelli]

Dan,

I tried uploading a cohort from a recent A/B test (1,780 unique user_id’s). The async validation took about 5 minutes to complete.

If I create a temporary table with the data in my CSV and run a join with the user table against a slave, the query to validate that these users exist takes about 400ms if I use user_id (primary key in enwiki.user) and about 3s using user_name (unique in enwiki.user).

What’s the reason why it takes so long to validate a cohort in the application?

Dario

Post by Dario Taraborelli
thanks Dan, this is awesome – I’ll give it a try this morning with some of the recent mobile cohorts.

Post by Dan Andreescu
Dear Wikimetrics users,
I've just deployed asynchronous cohort upload. This is feature #818: https://mingle.corp.wikimedia.org/projects/analytics/cards/818 and basically allows you to upload larger cohorts because validation is happening behind the scenes. I'll go over how the new functionality works here, and will rely on one of you to point me to the appropriate on-wiki place to update documentation.
* Remove Cohort. This is destructive and will remove this cohort from your list. Use this in case you made a mistake, uploaded the wrong file, etc.
* Validate Again. This will run validation again. One possible use for it is, let's say you upload a cohort with some *very* newly registered users. And because of replication lag to the labsdb databases, most of them come up invalid. You can then run validation again.
* Refresh. This just refreshes the status of the validation and will update the counts that show up below.
You will not have the "Create Report" option until validation is done. And when you do create a report, only valid users will be considered and used in the output.
One caveat. Validation is still slow. And the time limit for the asynchronous task is set to 1 hour. I have some ideas for making this faster by batching, and I can increase the time limit per task (but that has other repercussions). For now, just keep in mind that the theoretical maximum cohort size you should upload is roughly 18,000 users. I would love some feedback about whether it's ok to increase the time limit or if people want me to focus on making validation faster.
Dan
_______________________________________________
Wikimetrics mailing list
Wikimetrics at lists.wikimedia.org
https://lists.wikimedia.org/mailman/listinfo/wikimetrics

-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://lists.wikimedia.org/pipermail/wikimetrics/attachments/20131121/7014fd42/attachment.html>

[Steven Walling]

On Thu, Nov 21, 2013 at 9:57 AM, Dario Taraborelli <

Post by Dario Taraborelli
I tried uploading a cohort from a recent A/B test (1,780 unique
user_id’s). The async validation took about 5 minutes to complete.
If I create a temporary table with the data in my CSV and run a join with
the user table against a slave, the query to validate that these users
exist takes about 400ms if I use user_id (primary key in enwiki.user) and
about 3s using user_name (unique in enwiki.user).
What’s the reason why it takes so long to validate a cohort in the application?

My understanding is that this is due to Labs being slow compared to stat1?

[Dario Taraborelli]

I have no evidence that this is the case. A scan for the user table using the same fields/keys as the ones I used on the private slaves takes less than a second on tool labs.

Post by Dario Taraborelli
I tried uploading a cohort from a recent A/B test (1,780 unique user_id’s). The async validation took about 5 minutes to complete.
If I create a temporary table with the data in my CSV and run a join with the user table against a slave, the query to validate that these users exist takes about 400ms if I use user_id (primary key in enwiki.user) and about 3s using user_name (unique in enwiki.user).
What’s the reason why it takes so long to validate a cohort in the application?
My understanding is that this is due to Labs being slow compared to stat1?
--
Steven Walling,
Product Manager
https://wikimediafoundation.org/
_______________________________________________
Wikimetrics mailing list
Wikimetrics at lists.wikimedia.org
https://lists.wikimedia.org/mailman/listinfo/wikimetrics

-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://lists.wikimedia.org/pipermail/wikimetrics/attachments/20131121/fb90746f/attachment.html>

[Dan Andreescu]

Post by Steven Walling
On Thu, Nov 21, 2013 at 9:57 AM, Dario Taraborelli <

Post by Dario Taraborelli
I tried uploading a cohort from a recent A/B test (1,780 unique
user_id’s). The async validation took about 5 minutes to complete.
If I create a temporary table with the data in my CSV and run a join with
the user table against a slave, the query to validate that these users
exist takes about 400ms if I use user_id (primary key in enwiki.user) and
about 3s using user_name (unique in enwiki.user).
What’s the reason why it takes so long to validate a cohort in the application?

My understanding is that this is due to Labs being slow compared to stat1?

I don't think labs is that much slower though, we're talking orders of
magnitude here. So, I think the reason is that currently it's validating
one user at a time. Since for each record I have to check against a
potential user_id and user_name match, this takes forever.

Two ways to make it much faster:

* batch every X users and do a where user_id in (...) or user_name in (...)
query instead of checking each one
* create temporary tables just like Dario did

The problem is that cohorts can have users from multiple projects. That
makes both approaches harder, but should still be doable. The reason I
haven't done this yet is that when we scheduled 818 we broke out the
performance issue and agreed we'd work on it later. Sounds important
though, I'll look at it now.
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://lists.wikimedia.org/pipermail/wikimetrics/attachments/20131121/5884f57b/attachment.html>

[Dan Andreescu]

OK, I got 10k users to validate in about 30 seconds. Not instant but it
does have to do a bunch of duplicate checks, multi-project batching, etc.
Let me know how it works for you, and if there are any problems.

Post by Dan Andreescu

Post by Steven Walling
On Thu, Nov 21, 2013 at 9:57 AM, Dario Taraborelli <

Post by Dario Taraborelli
I tried uploading a cohort from a recent A/B test (1,780 unique
user_id’s). The async validation took about 5 minutes to complete.
If I create a temporary table with the data in my CSV and run a join
with the user table against a slave, the query to validate that these users
exist takes about 400ms if I use user_id (primary key in enwiki.user) and
about 3s using user_name (unique in enwiki.user).
What’s the reason why it takes so long to validate a cohort in the application?

My understanding is that this is due to Labs being slow compared to stat1?

I don't think labs is that much slower though, we're talking orders of
magnitude here. So, I think the reason is that currently it's validating
one user at a time. Since for each record I have to check against a
potential user_id and user_name match, this takes forever.
* batch every X users and do a where user_id in (...) or user_name in
(...) query instead of checking each one
* create temporary tables just like Dario did
The problem is that cohorts can have users from multiple projects. That
makes both approaches harder, but should still be doable. The reason I
haven't done this yet is that when we scheduled 818 we broke out the
performance issue and agreed we'd work on it later. Sounds important
though, I'll look at it now.

-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://lists.wikimedia.org/pipermail/wikimetrics/attachments/20131121/f345962d/attachment.html>

[Dario Taraborelli]

fantastic, is there any chance we could get an even better performance if we allowed users to specify the field type in the upload form (if it’s just user_ids, validation will be faster and the app doesn’t need to check every single entry for a valid user_name too). I understand that by design the application makes no assumption about the type of that field (and in fact it accepts a mix of user_id’s and user_names, correct)?

OK, I got 10k users to validate in about 30 seconds. Not instant but it does have to do a bunch of duplicate checks, multi-project batching, etc. Let me know how it works for you, and if there are any problems.
I tried uploading a cohort from a recent A/B test (1,780 unique user_id’s). The async validation took about 5 minutes to complete.
If I create a temporary table with the data in my CSV and run a join with the user table against a slave, the query to validate that these users exist takes about 400ms if I use user_id (primary key in enwiki.user) and about 3s using user_name (unique in enwiki.user).
What’s the reason why it takes so long to validate a cohort in the application?
My understanding is that this is due to Labs being slow compared to stat1?
I don't think labs is that much slower though, we're talking orders of magnitude here. So, I think the reason is that currently it's validating one user at a time. Since for each record I have to check against a potential user_id and user_name match, this takes forever.
* batch every X users and do a where user_id in (...) or user_name in (...) query instead of checking each one
* create temporary tables just like Dario did
The problem is that cohorts can have users from multiple projects. That makes both approaches harder, but should still be doable. The reason I haven't done this yet is that when we scheduled 818 we broke out the performance issue and agreed we'd work on it later. Sounds important though, I'll look at it now.
_______________________________________________
Wikimetrics mailing list
Wikimetrics at lists.wikimedia.org
https://lists.wikimedia.org/mailman/listinfo/wikimetrics

-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://lists.wikimedia.org/pipermail/wikimetrics/attachments/20131121/7bbd7999/attachment-0001.html>

[Dan Andreescu]

On Thu, Nov 21, 2013 at 6:00 PM, Dario Taraborelli <

Post by Dario Taraborelli
fantastic, is there any chance we could get an even better performance if
we allowed users to specify the field type in the upload form (if it’s just
user_ids, validation will be faster and the app doesn’t need to check every
single entry for a valid user_name too). I understand that by design the
application makes no assumption about the type of that field (and in fact
it accepts a mix of user_id’s and user_names, correct)?

absolutely, it would run up to 2x faster if the file was all user_ids and
the user specified that up front. But currently, yes, you can mix user_ids
and user_names
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://lists.wikimedia.org/pipermail/wikimetrics/attachments/20131121/05b2dee5/attachment.html>

[Dario Taraborelli]

is there any foreseeable use case for mixed cohorts? If not, it sounds like this would be a useful enhancement.

Post by Dario Taraborelli
fantastic, is there any chance we could get an even better performance if we allowed users to specify the field type in the upload form (if it’s just user_ids, validation will be faster and the app doesn’t need to check every single entry for a valid user_name too). I understand that by design the application makes no assumption about the type of that field (and in fact it accepts a mix of user_id’s and user_names, correct)?
absolutely, it would run up to 2x faster if the file was all user_ids and the user specified that up front. But currently, yes, you can mix user_ids and user_names
_______________________________________________
Wikimetrics mailing list
Wikimetrics at lists.wikimedia.org
https://lists.wikimedia.org/mailman/listinfo/wikimetrics

-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://lists.wikimedia.org/pipermail/wikimetrics/attachments/20131121/46dd24d8/attachment.html>

[Dan Andreescu]

Post by Dario Taraborelli
is there any foreseeable use case for mixed cohorts? If not, it sounds
like this would be a useful enhancement.

I await the prioritization gods to tell me what to do :) Personally, I
think dynamic cohorts and timeseries for all metrics might be more
important.
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://lists.wikimedia.org/pipermail/wikimetrics/attachments/20131121/53b9f94e/attachment.html>

[Edward Galvez]

Thanks for this! Just did a cohort of 15K and it worked fine. I
unintentionally hit the "back" button, but after clicking "forward" and my
cohort was validated, not even 10 seconds later.

Also, I can't seem to find the place that listed which users were not
valid. Did we lose that ability?

- E

Post by Dan Andreescu
On Thu, Nov 21, 2013 at 6:00 PM, Dario Taraborelli <

Post by Dario Taraborelli
fantastic, is there any chance we could get an even better performance if
we allowed users to specify the field type in the upload form (if it’s just
user_ids, validation will be faster and the app doesn’t need to check every
single entry for a valid user_name too). I understand that by design the
application makes no assumption about the type of that field (and in fact
it accepts a mix of user_id’s and user_names, correct)?

absolutely, it would run up to 2x faster if the file was all user_ids and
the user specified that up front. But currently, yes, you can mix user_ids
and user_names
_______________________________________________
Wikimetrics mailing list
Wikimetrics at lists.wikimedia.org
https://lists.wikimedia.org/mailman/listinfo/wikimetrics

-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://lists.wikimedia.org/pipermail/wikimetrics/attachments/20131121/17e6f3c9/attachment.html>

[Edward Galvez]

Also! Just to introduce myself, I'm one of the interns with the Program
Evaluation & Design team - thus this upgrade is very timely. Thank you!

Post by Edward Galvez
Thanks for this! Just did a cohort of 15K and it worked fine. I
unintentionally hit the "back" button, but after clicking "forward" and my
cohort was validated, not even 10 seconds later.
Also, I can't seem to find the place that listed which users were not
valid. Did we lose that ability?
- E

Post by Dan Andreescu
On Thu, Nov 21, 2013 at 6:00 PM, Dario Taraborelli <

Post by Dario Taraborelli
fantastic, is there any chance we could get an even better performance
if we allowed users to specify the field type in the upload form (if it’s
just user_ids, validation will be faster and the app doesn’t need to check
every single entry for a valid user_name too). I understand that by design
the application makes no assumption about the type of that field (and in
fact it accepts a mix of user_id’s and user_names, correct)?

absolutely, it would run up to 2x faster if the file was all user_ids and
the user specified that up front. But currently, yes, you can mix user_ids
and user_names
_______________________________________________
Wikimetrics mailing list
Wikimetrics at lists.wikimedia.org
https://lists.wikimedia.org/mailman/listinfo/wikimetrics

-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://lists.wikimedia.org/pipermail/wikimetrics/attachments/20131121/ca3b8deb/attachment.html>

[Dan Andreescu]

Hi Edward.  Yes, we temporarily lost the UI that shows what users are invalid.  I wasn't sure what exactly people needed here so I didn't hazard a guess.   The data is all there though and I can easily show you invalid users and invalid reasons for your cohort.




I just maybe need you and someone else to say how you'd like it to work and I can whip up a view for it tomorrow.  Dario, any opinion on how invalid users should be displayed?  The only weird part now is that you can't upload again.  You'd have to delete the whole cohort and start over...

—
Sent from Mailbox for iPhone

On Thu, Nov 21, 2013 at 7:20 PM, Edward Galvez <egalvez at wikimedia.org>

Post by Edward Galvez
Also! Just to introduce myself, I'm one of the interns with the Program
Evaluation & Design team - thus this upgrade is very timely. Thank you!

Post by Edward Galvez
Thanks for this! Just did a cohort of 15K and it worked fine. I
unintentionally hit the "back" button, but after clicking "forward" and my
cohort was validated, not even 10 seconds later.
Also, I can't seem to find the place that listed which users were not
valid. Did we lose that ability?
- E

Post by Dan Andreescu
On Thu, Nov 21, 2013 at 6:00 PM, Dario Taraborelli <

Post by Dario Taraborelli
fantastic, is there any chance we could get an even better performance
if we allowed users to specify the field type in the upload form (if it’s
just user_ids, validation will be faster and the app doesn’t need to check
every single entry for a valid user_name too). I understand that by design
the application makes no assumption about the type of that field (and in
fact it accepts a mix of user_id’s and user_names, correct)?

absolutely, it would run up to 2x faster if the file was all user_ids and
the user specified that up front. But currently, yes, you can mix user_ids
and user_names
_______________________________________________
Wikimetrics mailing list
Wikimetrics at lists.wikimedia.org
https://lists.wikimedia.org/mailman/listinfo/wikimetrics

-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://lists.wikimedia.org/pipermail/wikimetrics/attachments/20131121/c922b4d8/attachment-0001.html>

[Dario Taraborelli]

I (and by extension other people in the research team) will probably only ever use user_ids (which we know in advance are valid), so it’s probably best to ask Program Evauation folks or community members who may rely on usernames.

Hi Edward. Yes, we temporarily lost the UI that shows what users are invalid. I wasn't sure what exactly people needed here so I didn't hazard a guess. The data is all there though and I can easily show you invalid users and invalid reasons for your cohort.
I just maybe need you and someone else to say how you'd like it to work and I can whip up a view for it tomorrow. Dario, any opinion on how invalid users should be displayed? The only weird part now is that you can't upload again. You'd have to delete the whole cohort and start over...
—
Sent from Mailbox for iPhone
Also! Just to introduce myself, I'm one of the interns with the Program Evaluation & Design team - thus this upgrade is very timely. Thank you!
Thanks for this! Just did a cohort of 15K and it worked fine. I unintentionally hit the "back" button, but after clicking "forward" and my cohort was validated, not even 10 seconds later.
Also, I can't seem to find the place that listed which users were not valid. Did we lose that ability?
- E
fantastic, is there any chance we could get an even better performance if we allowed users to specify the field type in the upload form (if it’s just user_ids, validation will be faster and the app doesn’t need to check every single entry for a valid user_name too). I understand that by design the application makes no assumption about the type of that field (and in fact it accepts a mix of user_id’s and user_names, correct)?
absolutely, it would run up to 2x faster if the file was all user_ids and the user specified that up front. But currently, yes, you can mix user_ids and user_names
_______________________________________________
Wikimetrics mailing list
Wikimetrics at lists.wikimedia.org
https://lists.wikimedia.org/mailman/listinfo/wikimetrics
_______________________________________________
Wikimetrics mailing list
Wikimetrics at lists.wikimedia.org
https://lists.wikimedia.org/mailman/listinfo/wikimetrics

-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://lists.wikimedia.org/pipermail/wikimetrics/attachments/20131121/28732f04/attachment.html>

[Jaime Anstee]

Yes, we need user names and I can imagine some cases for potential mixed cohorts, but not sure about prevalence - Jaime

Post by Dario Taraborelli
I (and by extension other people in the research team) will probably only ever use user_ids (which we know in advance are valid), so it’s probably best to ask Program Evauation folks or community members who may rely on usernames.

Hi Edward. Yes, we temporarily lost the UI that shows what users are invalid. I wasn't sure what exactly people needed here so I didn't hazard a guess. The data is all there though and I can easily show you invalid users and invalid reasons for your cohort.
I just maybe need you and someone else to say how you'd like it to work and I can whip up a view for it tomorrow. Dario, any opinion on how invalid users should be displayed? The only weird part now is that you can't upload again. You'd have to delete the whole cohort and start over...
—
Sent from Mailbox for iPhone

Also! Just to introduce myself, I'm one of the interns with the Program Evaluation & Design team - thus this upgrade is very timely. Thank you!

Thanks for this! Just did a cohort of 15K and it worked fine. I unintentionally hit the "back" button, but after clicking "forward" and my cohort was validated, not even 10 seconds later.
Also, I can't seem to find the place that listed which users were not valid. Did we lose that ability?
- E

Post by Dario Taraborelli

Post by Dario Taraborelli
fantastic, is there any chance we could get an even better performance if we allowed users to specify the field type in the upload form (if it’s just user_ids, validation will be faster and the app doesn’t need to check every single entry for a valid user_name too). I understand that by design the application makes no assumption about the type of that field (and in fact it accepts a mix of user_id’s and user_names, correct)?

absolutely, it would run up to 2x faster if the file was all user_ids and the user specified that up front. But currently, yes, you can mix user_ids and user_names
_______________________________________________
Wikimetrics mailing list
Wikimetrics at lists.wikimedia.org
https://lists.wikimedia.org/mailman/listinfo/wikimetrics

_______________________________________________
Wikimetrics mailing list
Wikimetrics at lists.wikimedia.org
https://lists.wikimedia.org/mailman/listinfo/wikimetrics

_______________________________________________
Wikimetrics mailing list
Wikimetrics at lists.wikimedia.org
https://lists.wikimedia.org/mailman/listinfo/wikimetrics

-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://lists.wikimedia.org/pipermail/wikimetrics/attachments/20131121/1a69df0b/attachment.html>

[Dan Andreescu]

Post by Edward Galvez
Thanks for this! Just did a cohort of 15K and it worked fine. I
unintentionally hit the "back" button, but after clicking "forward" and my
cohort was validated, not even 10 seconds later.
Also, I can't seem to find the place that listed which users were not
valid. Did we lose that ability?

OK Edward, I added a little link to see the invalid users for your
validated cohort. You just click on the "X are invalid" text on your
cohort's tab. This will list the value it tried to validate and the reason
it thought it was invalid.

Dan
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://lists.wikimedia.org/pipermail/wikimetrics/attachments/20131123/9fd1ef2a/attachment.html>