Dan Andreescu
2013-11-16 10:02:30 UTC
The forwarded message is very relevant for wikimetrics. Â I have some basic backups but we really should migrate the service to production. Â There are a few things to fix before we do so, but if we don't want any interruptions or data loss, this *has* to happen.
â
Sent from Mailbox for iPhone
---------- Forwarded message ----------
From: "Andrew Bogott" <abogott at wikimedia.org>
Date: Fri, Nov 15, 2013 at 2:29 PM
Subject: [Wikitech-l] Labs datacenter migration
To: "A list for announcements and discussion related to the Wikimedia Labs project." <labs-l at lists.wikimedia.org>, "Wikimedia developers" <wikitech-l at lists.wikimedia.org>
An HTML attachment was scrubbed...
URL: <http://lists.wikimedia.org/pipermail/wikimetrics/attachments/20131116/60bf280e/attachment.html>
â
Sent from Mailbox for iPhone
---------- Forwarded message ----------
From: "Andrew Bogott" <abogott at wikimedia.org>
Date: Fri, Nov 15, 2013 at 2:29 PM
Subject: [Wikitech-l] Labs datacenter migration
To: "A list for announcements and discussion related to the Wikimedia Labs project." <labs-l at lists.wikimedia.org>, "Wikimedia developers" <wikitech-l at lists.wikimedia.org>
Almost a year ago, the Wikimedia Foundation migrated most of our
services from our old data center in Tampa to the new one in Ashburn
[1]. In the next couple of months Labs and Tool Labs will be following
suit -- we expect to have everything moved to Ashburn by mid-January at
the latest.
This move will provide some immediate benefits (lower latency with
production, quicker database replication) and many long-term benefits
(better stability, happier Operations staff). We don't yet have a
specific timeline for stages of the migration, but there are a few
things you can do now to help us prepare for the change and to bolster
your projects against possible disruption.
1) Subscribe to Labs-l, and read it. [2] Labs-l is low-volume, and
future migration announcements may not be sent to other lists.
2) Tool Labs users: As long as your tools are properly managed by the
grid engine and can survive stops and restarts, the migration will be
quite painless. If your tools aren't, or can't... fix them :)
3) Labs project admins: Clean up old projects and instances. If you
have instances that are no longer of interest, delete them. If you know
of entire projects that are no longer in use, please contact me directly
and I'll mop up.
4) Labs instance owners: Make sure that puppet is running properly on
your instances. If '$sudo puppetd -tv' produces any red lines, then fix
them or contact me for help with fixing. When instances move to the new
data center we'll be relying on puppet to update location-specific
settings, so instances without puppet may not survive the move. If your
instance uses self-hosting puppet (via puppetmaster::self or
role::puppet::self) then you will also need to update your local puppet
repo. [3]
5) All Labs users: if you have valuable data residing on local instance
storage, start backing it up to shared storage in /data/project. You
should be doing this anyway -- no instance is safe from catastrophe, ever.
6) If your project or tool generates log files, have a look at purging
old log data. The last time we did a data migration there was at least
one terabyte-sized logfile that really gummed up the works.
Updates about this change will be posted to this list as soon as we
know about them. Any potential downtime will be announced well in
advance. In the meantime, don't hesitate to ask questions about the
above steps on IRC or the mailing list.
-Andrew
[1]
https://blog.wikimedia.org/2013/01/19/wikimedia-sites-move-to-primary-data-center-in-ashburn-virginia/
[2] https://lists.wikimedia.org/mailman/listinfo/labs-l
[3] https://wikitech.wikimedia.org/wiki/Help:Self-hosted_puppetmaster#FAQ
_______________________________________________
Wikitech-l mailing list
Wikitech-l at lists.wikimedia.org
https://lists.wikimedia.org/mailman/listinfo/wikitech-l
-------------- next part --------------services from our old data center in Tampa to the new one in Ashburn
[1]. In the next couple of months Labs and Tool Labs will be following
suit -- we expect to have everything moved to Ashburn by mid-January at
the latest.
This move will provide some immediate benefits (lower latency with
production, quicker database replication) and many long-term benefits
(better stability, happier Operations staff). We don't yet have a
specific timeline for stages of the migration, but there are a few
things you can do now to help us prepare for the change and to bolster
your projects against possible disruption.
1) Subscribe to Labs-l, and read it. [2] Labs-l is low-volume, and
future migration announcements may not be sent to other lists.
2) Tool Labs users: As long as your tools are properly managed by the
grid engine and can survive stops and restarts, the migration will be
quite painless. If your tools aren't, or can't... fix them :)
3) Labs project admins: Clean up old projects and instances. If you
have instances that are no longer of interest, delete them. If you know
of entire projects that are no longer in use, please contact me directly
and I'll mop up.
4) Labs instance owners: Make sure that puppet is running properly on
your instances. If '$sudo puppetd -tv' produces any red lines, then fix
them or contact me for help with fixing. When instances move to the new
data center we'll be relying on puppet to update location-specific
settings, so instances without puppet may not survive the move. If your
instance uses self-hosting puppet (via puppetmaster::self or
role::puppet::self) then you will also need to update your local puppet
repo. [3]
5) All Labs users: if you have valuable data residing on local instance
storage, start backing it up to shared storage in /data/project. You
should be doing this anyway -- no instance is safe from catastrophe, ever.
6) If your project or tool generates log files, have a look at purging
old log data. The last time we did a data migration there was at least
one terabyte-sized logfile that really gummed up the works.
Updates about this change will be posted to this list as soon as we
know about them. Any potential downtime will be announced well in
advance. In the meantime, don't hesitate to ask questions about the
above steps on IRC or the mailing list.
-Andrew
[1]
https://blog.wikimedia.org/2013/01/19/wikimedia-sites-move-to-primary-data-center-in-ashburn-virginia/
[2] https://lists.wikimedia.org/mailman/listinfo/labs-l
[3] https://wikitech.wikimedia.org/wiki/Help:Self-hosted_puppetmaster#FAQ
_______________________________________________
Wikitech-l mailing list
Wikitech-l at lists.wikimedia.org
https://lists.wikimedia.org/mailman/listinfo/wikitech-l
An HTML attachment was scrubbed...
URL: <http://lists.wikimedia.org/pipermail/wikimetrics/attachments/20131116/60bf280e/attachment.html>