slides

Who Moved My Module?
®
®
1
About Me
• Yoav Landman
- JFrog’s CTO and Founder
- Creator of the Artifactory Project
- 10+ years experience in commercial enterprise
build and development environments
QCon London 2011
2
Agenda
• Modular software development
• Why manage modules
• Using modules in release management
- Release strategies and models
- Common release issues
QCon London 2011
3
Going modular
http://www.flickr.com/photos/jemsweb/4363548805
QCon London 2011
4
Java modules
• Java IDEs - project modules introduced
circa 2004/2005
• Maven, Gradle, Ivy, Jigsaw, OSGi...
• CI Servers
- Cascading builds according to dependencies
- Per module results
• A reality
QCon London 2011
5
Module key role players
build
environment/
tools
use
module
system
resolve
shared module
repositories
use
runtime
environment
manage, resolve
module
database
QCon London 2011
6
key role players - real world
compile,
package...
(maven, gradle, ivy
+CI servers)
use
resolve,
publish
use
OSGi, Jigsaw,
Web Start...
resolve
(maven, gradle, ivy)
binary
repositories
manage, resolve
local
repository/
cache
QCon London 2011
7
Demo Time
Meet Artifactory
QCon London 2011
8
A Binary Repository
• A shared place for binaries
- 3rd party and local artifacts
• Much more than a passive storage
• Critical for CI and ALM
QCon London 2011
9
What’s wrong with my...
• Subversion
• Apache file server/webdav
• Shared file system
• DVD
• Disk-on-key, tape, drawer...
QCon London 2011
10
Can your shared file system/VCS
•
•
•
•
•
•
•
•
•
QCon London 2011
Download and cache remote artifacts?
Promote artifacts during release?
Control licenses used in your software?
Track and preserve artifacts used by CI builds?
Enforce module CRUD security easily?
Track changes made to artifacts?
Automatically clean up integration garbage?
Manage artifacts with powerful searches?
Expose powerful REST API?
11
But I can always rebuild my modules
from source!
http://www.flickr.com/photos/a-culinary-photo-journal/3134396770
QCon London 2011
12
Rebuilding from VCS
• Expensive & time consuming
- Sometimes non-practical (resources)
- Disaster recovery time
• Dynamic
- Properties, version ranges, etc.
- Reliance on a specific environment
QCon London 2011
13
The Artifact Repository Ecosystem
ion
t
a
l
o
s
i
build
replica
ti
VCS
onModules/Artifacts/Libraries
ns
o
i
s
s
n
promot
d
e
l
i
t
u
ex
sb
u
r
o
e
u
i
s
o
n
i
u
t
n
n
o
c
conten
Remote
t-filteri Repository
s
e
ng
Repositories
search
securit
y mgm
license c
t.
ontrol
g
n
i
k
c
a
r
t
build
n
o
i
s
i
v
o
r
p
host &
ss
e
c
c
a
T
RES
QCon London 2011
proxy
& cach
metadat
a and ta
gging
e
14
Artifactory
• Advanced Binary Repository Manager
• First searchable, web-driven repository
manager (2006)
• Over120,000 downloads (Feb 2011)
• OSS, Pro & Cloud versions
- jfrog.org | jfrog.com | artifactoryonline.com
• Shaping the binary repository arena
QCon London 2011
r
e
v
e
N
!
k
c
a
b
k
o
o
l
15
Demo Time
on-boarding & searching
QCon London 2011
16
Artifact Build Integration Platform
http://www.flickr.com/photos/skrb/1326663872
QCon London 2011
17
Artifact Build Integration Platform
#1
®
#1-1
#1-2
#3-2
#3-1
#2-2
#2
#3-3
#2-1
#3
QCon London 2011
18
Across-the-board integration
QCon London 2011
19
Demo Time
Build integration
QCon London 2011
20
The release pipeline
• Multiple steps towards declaring a release
• Check points for advancing the release flow
modules
modules
modules
modules
Source: Agile ALM, Michael Hüttermann, Manning Publications Co.
QCon London 2011
21
The release pipeline model
A
B
C
GM
Linear
A
Non-linear
B
C
GM
QCon London 2011
22
The release pipeline - YMMV
http://www.flickr.com/photos/jaxxon/3335409285
QCon London 2011
23
Typical release issues
• Picking a source/release strategy
• Release build Isolation
• Change release status without recompilation
- Dynamic module descriptors
• Avoiding license “creep”
QCon London 2011
24
Source/release strategies
• Release-to-branch
• M2 Release plugin
• Artifactory build-integration release
QCon London 2011
25
Merge-to-branch
trunk
job#1: build/test
release
branch
tag
merge into branch
job#2: build/test
release tag
tag a release
QCon London 2011
26
Jenkins M2 release plugin
1.
Check out latest VCS revision
2.
Compile & test
3.
Change POMs to next release version
4.
Compile & test
5.
Commit new POMs (!)
6.
Create a VCS tag from WC
7.
Change POMs to next dev version (++-SNAPSHOT)
8.
Commit new POMs
9.
Check out previously created VCS tag
http://www.flickr.com/photos/rsdio/3642425935
10. Compile & test
11. Publish binaries to repository
QCon London 2011
27
Jenkins M2 release plugin
• Attempts to reuse a user-operated plugin in
a CI environment
- Uses separate external Maven process
- Uses separate external VCS (svn auth)
• No good rollback
- End users may like CI’s VCS permissions
QCon London 2011
28
Artifactory plugin
1. Change POMs to next release version
2. Compile & test
3. Publish binaries to repository
4. Create a VCS tag from WC
5. Change POMs to next dev version
(version++-SNAPSHOT)
6. Commit new POMs
QCon London 2011
29
Releasing with the Artifactory plugin
• Fast
• Easy automatic rollback
- E.g. tag removal
• Closely integrated with CI server
- Reuses Maven and VCS definitions
• Support for Ivy and Gradle + other CI
servers in the works
QCon London 2011
30
Demo Time
Release & promotion
QCon London 2011
31
Release build isolation
• Cascading jobs - tests run longer
• A downstream job may take wrong
artifacts put by an upstream
job
• No isolation - the build integration
silo is broken
http://www.flickr.com/photos/sharynmorrow/5961475
QCon London 2011
32
Keeping integration private
time
job-a
#12
job-b
#4
job-a
#13
job-c
X
®
A#12
B#4
C#2
???
A#13
QCon London 2011
33
Isolating resolution
• Deploy as many integration revs in parallel
• Each build resolves only the artifacts in the
build chain it is part of
• Achieved with CI artifact painting and
matrix params for resolution:
http://.../artifactory/jfrog/app-1.0-SNAPSHOT.jar;build.root=j-1025
QCon London 2011
34
Demo Time
Release isolation/private builds
QCon London 2011
35
Avoiding recompilation on release
Product
Binaries
v1.1-SNAP
v1.1-SNAP
v1.1-SNAP
Qualified
v1.1-SNAP
v1.1-SNAP
v1.1-SNAP
v1.1-SNAP
•
•
v1.1-SNAP
A collection of release-qualified integration packages
Lack of release status
-
•
QCon London 2011
No way to express that without changing pom/ivy descriptors
Resolution for packaging relies on descriptors
36
Resolution with integration versions
•
Ivy/Gradle
-
Integration revisions same as any release revision (1.0-782)
-
By default all dynamic versions are resolved and replaced in
delivered modules
-
Can be used in a release package
•
Maven
-
Published artifacts can contain dynamic dependencies
(SNAPSHOT or ranges) + unresolved properties
-
Very hard to keep reproducibility
QCon London 2011
37
License “creep”
• Verify no unwanted 3rd party is packaged
into the release
• Discover and apply license information
continuously!
- Information is there
- Discover and act early
QCon London 2011
38
License violation detection
QCon London 2011
39
Demo Time
Detecting license violations
QCon London 2011
40
Summary
• Why do we need a binary repository
• Smart modules hub that plays key roles in
- Continuous integration and delivery
- Release management
• Really changes how we work and think
about binaries
• Constantly evolving
QCon London 2011
41
Thank you! Questions?
QCon London 2011
42