In this example of VXVM 4.0 on a Solaris 8 system, an array was temporarily unavailable, causing problems with a file system whose two plexes resided on the array.
bash-2.03# cd /files04
bash: cd: /files04: I/O error
The volume was in DISABLED ACTIVE state, and both plexes were in DISABLED RECOVER state.
v vol04 - DISABLED ACTIVE 29360128 SELECT - fsgen
pl vol04-01 vol04 DISABLED RECOVER 29367434 STRIPE 3/128 RW
sd appsdg01-04 vol04-01 cs_array07-f0 8392167 2797389 0/0 c1t0d0 ENA
sd appsdg07-01 vol04-01 cs_array03-f2 0 5594778 0/2797389 c4t2d0 ENA
sd appsdg07-04 vol04-01 cs_array03-f2 11189556 1396899 0/8392167 c4t2d0 ENA
sd appsdg02-04 vol04-01 cs_array07-f1 8392167 2797389 1/0 c1t1d0 ENA
sd appsdg10-02 vol04-01 cs_array06-f1 2797389 5594778 1/2797389 c5t1d0 ENA
sd appsdg10-05 vol04-01 cs_array06-f1 13986945 1396899 1/8392167 c5t1d0 ENA
sd appsdg03-04 vol04-01 cs_array07-f2 8392167 2797389 2/0 c1t2d0 ENA
sd appsdg11-02 vol04-01 cs_array06-f2 8392167 6991677 2/2797389 c5t2d0 ENA
pl vol04-02 vol04 DISABLED RECOVER 29367434 STRIPE 3/128 RW
sd appsdg04-02 vol04-02 cs_array07-f3 2797389 2797389 0/0 c1t3d0 ENA
sd appsdg04-05 vol04-02 cs_array07-f3 0 2797389 0/2797389 c1t3d0 ENA
sd appsdg04-06 vol04-02 cs_array07-f3 16784334 894159 0/5594778 c1t3d0 ENA
sd appsdg14-02 vol04-02 cs_array07-f6 12586455 3300129 0/6488937 c1t6d0 ENA
sd appsdg12-03 vol04-02 cs_array06-f3 5594778 2797389 1/0 c5t3d0 ENA
sd appsdg13-02 vol04-02 cs_array07-f4 12586455 5092038 1/2797389 c1t4d0 ENA
sd appsdg12-02 vol04-02 cs_array06-f3 16784334 894159 1/7889427 c5t3d0 ENA
sd appsdg05-02 vol04-02 cs_array03-f0 12586455 1005480 1/8783586 c4t0d0 ENA
sd appsdg09-02 vol04-02 cs_array06-f0 2797389 8392167 2/0 c5t0d0 ENA
sd appsdg09-06 vol04-02 cs_array06-f0 3591 1396899 2/8392167 c5t0d0 ENA
We confirmed that the storage array was available to the operating system.
# luxadm probe
Found Enclosure(s):
...
SENA Name:cs_array06 Node WWN:5080020000038ba8
Logical Path:/dev/es/ses6
Logical Path:/dev/es/ses7
# luxadm display cs_array06
SENA
DISK STATUS
SLOT FRONT DISKS (Node WWN) REAR DISKS (Node WWN)
0 On (O.K.) 2000002037094289 On (O.K.) 200000203709422e
1 On (O.K.) 2000002037093aaf On (O.K.) 2000002037094220
2 On (O.K.) 200000203709410b On (O.K.) 2000002037093ddd
3 On (O.K.) 2000002037094254 On (O.K.) 200000203709422b
4 On (O.K.) 20000020370940da On (O.K.) 2000002037094247
5 Not Installed Not Installed
6 On (O.K.) 2000002037093df0 On (O.K.) 200000203709383f
Next, we reattached the disks to the disk group they were in. You may want to run vxreattach -c diskname to check if a reattach is possible before attempting to reattach the disks.
# vxdisk list
...
- - cs_array06-f0 appsdg failed was:c5t0d0s2
- - cs_array06-f1 appsdg failed was:c5t1d0s2
- - cs_array06-f2 appsdg failed was:c5t2d0s2
- - cs_array06-f3 appsdg failed was:c5t3d0s2
- - cs_array06-r4 appsdg failed spare was:c5t20d0s2
- - cs_array06-f4 appsdg failed was:c5t4d0s2
# cd /usr/lib/vxvm/bin
# ./vxreattach c5t0d0s2
# ./vxreattach c5t1d0s2
# ./vxreattach c5t2d0s2
# ./vxreattach c5t3d0s2
# ./vxreattach c5t20d0s2
# ./vxreattach c5t4d0s2
We then followed the "Recovering an Unstartable Volume with a Disabled Plex in the RECOVER State" procedure in the Volume Manager Troubleshooting Guide.
1. Force plex vol04-01 into the OFFLINE state.
# vxmend -g appsdg -o force off vol04-01
2. Place plex vol04-01 into the STALE state.
# vxmend -g appsdg on vol04-01
3. There are no other clean plexes in the volume, so make plex vol04-01 DISABLED and CLEAN.
# vxmend -g appsdg fix clean vol04-01
4. Start the volume, and perform resynchronization of the plexes in the background.
# vxvol -g appsdg -o bg start vol04
At this point, the file system is unmounted, checked for file system consistency, and remounted.
# umount /files04
# mount /files04
UX:vxfs mount: ERROR: V-3-21268: /dev/vx/dsk/appsdg/vol04 is corrupted. needs checking
# fsck -F vxfs /dev/vx/rdsk/appsdg/vol04
log replay in progress
replay complete - marking super-block as CLEAN
# mount /files04
Sunday, March 7, 2010
Veritas notes
The following notes are for Veritas Volume Manager 3.2 for Solaris.
"vxvm:vxconfigd: ERROR: enable failed: Error in disk group configuration copies
Disk group has no valid configuration copies; transactions are disabled."
When receiving this error during system boot and when running vxinstall, follow the steps detailed in http://www.eng.auburn.edu/pub/mail-lists/veritas-users.May99/msg00048.html
In my case, the rootdg configuration was apparently corrupted. After issuing touch /etc/vx/reconfig.d/state.d/install-db and rebooting the machine, I was able to run vxinstall.
vxvm:vxdg: ERROR: Disk group disk_group: import failed: Disk group has no valid configuration copies"
This error can occur when attempting to import a disk group that was configured using a later version of VxVM. In this case, the disk group was configured with VxVM 3.2, but VxVM 3.1.1 was installed.
# pkginfo -l VRTSvxvm
PKGINST: VRTSvxvm
NAME: VERITAS Volume Manager, Binaries
CATEGORY: system
ARCH: sparc
VERSION: 3.1.1,REV=01.30.2001.22.21
Upgrading to at least the same version of VxVM used to configure the disk group will allow the disk group to be imported.
"ld.so.1: vxconfigd: fatal: libdevid.so.1: open failed: No such file or directory"
With Solaris 8 and VxVM 3.2, the shared library libdevid.so.1 does not get copied to /etc/vx/slib after installing Veritas. If you do not manually copy this shared library to /etc/vx/slib, your system will not boot. Follow these steps to make your system bootable:
1. Boot off a CD-ROM.
2. Mount your root and usr file systems.
3. Copy /usr/lib/libdevid.so.1 to /etc/vx/slib
4. Unmount your root and usr file systems and reboot.
More information:
http://marc.theaimsgroup.com/?l=veritas-vx&m=102636855529467&w=2
Clearing device locks
To clear a device lock, use the vxdisk clearimport command:
vxdisk clearimport devicename
ex. vxdisk clearimport c0t1d0
Using a Sun StorEdge A5000 disk array with Veritas
Make sure the array(s) are recognized by the operating system.
# luxadm probe
Found Enclosure(s):
SENA Name:a1 Node WWN:50800200000276e0
Logical Path:/dev/es/ses2
Logical Path:/dev/es/ses7
SENA Name:a2 Node WWN:5080020000028020
Logical Path:/dev/es/ses3
Logical Path:/dev/es/ses6
SENA Name:a0 Node WWN:5080020000026f38
Logical Path:/dev/es/ses4
Logical Path:/dev/es/ses5
SENA Name:a3 Node WWN:5080020000027060
Logical Path:/dev/es/ses8
Logical Path:/dev/es/ses9
Run Veritas' device discovery program.
# vxdctl enable
Determining maximum size of a volume
vxassist [ -g diskgroup ] maxsize layout=layout [attributes]
Example:
vxassist -g datadg maxsize layout=concat
layout may be concat, mirror, raid5, mirror-stripe, or stripe-mirror.
Veritas disk requirements
Disks managed by VxVM must have (1) two free partitions and (2) 2048 sectors of free space. The prtvtoc command displays how many sectors are in a disk cylinder:
# prtvtoc /dev/rdsk/c0t86d0s2
* /dev/rdsk/c0t86d0s2 partition map
*
* Dimensions:
* 512 bytes/sector
* 133 sectors/track
* 27 tracks/cylinder
* 3591 sectors/cylinder
* 4926 cylinders
* 4924 accessible cylinders
In this example, leave at least 1 cylinder free in your disk layout to allow for VxVM. If the disk is a boot disk, VxVM can shrink the swap partition to create space for VxVM's configuration data, but two free slices are essential for encapsulation.
Creating a volume with vxassist
ex.
# vxassist -g datadg maxsize
Maximum volume size: 35356672 (17264Mb)
# vxassist -g datadg make volume 35356672
Create the vxfs file system:
# mkfs -F vxfs /dev/vx/rdsk/datadg/db_backups
version 4 layout
35356672 sectors, 17678336 blocks of size 1024, log size 16384 blocks
unlimited inodes, largefiles not supported
17678336 data blocks, 17657432 free data blocks
540 allocation units of 32768 blocks, 32768 data blocks
last allocation unit has 16384 data blocks
Create the mount point:
# mkdir /db_backups
Mount the vxfs file system:
# mount -F vxfs /dev/vx/dsk/datadg/db_backups /db_backups
Add an /etc/vfstab entry to mount the file system after a reboot.
Replacing a failed disk
After replacing a failed disk in a SENA, make sure to run vxdctl enable for device discovery. Otherwise, you may encounter vxdmpadm errors:
Initialization of disk device c1t74d0 failed.
Error: vxvm:vxdmpadm: ERROR: Error in ioctl/open
vxdmpadm: No such file or directory
vxvm:vxdmpadm: ERROR: Invalid da_name
vxvm:vxdmpadm: ERROR: Invalid da_name
vxdisksetup: c1t74d0: Device address must be of the form cCtTdD or mcCtTdD where
C = host bus adapter controller number
T = target device controller number, if used
D = logical unit (disk) number within target device controller
# vxdisk list c1t74d0s2
Device: c1t74d0s2
devicetag: c1t74d0
type: sliced
flags: online error private autoconfig
errno: Device path not valid
Multipathing information:
numpaths: 2
c1t74d0s2 state=disabled
c5t74d0s2 state=disabled
When replacing a failed internal disk on a Sun E450 running Solaris 8, I had to spin the disk down using ssaadm stop /dev/rdsk/cxtxdxs2as the vxdiskadm's "Disable (offline) a disk device" did not seem to spin the disk down. If you are using a Sun system with FC-AL devices, you will want to use the luxadm command.
After replacing the disk, I enabled device discovery with vxdctl enable and un-relocated the failed subdisks back to this disk using /usr/lib/vxvm/bin/vxunreloc -g disk_group replaced_disk.
Adding additional users to VxVM electronic mail notifications
By default, VxVM sends electronic mail to the root user when failures are detected and hot-relocation is being performed. To notify additional users,
1. Edit /etc/init.d/vxvm-recover
2. Change the line containing vxrelocd root & to vxrelocd root user1 user2 ... &
This will preserve the change across system reboot.
3. To have the change take effect immediately, make sure that hot-relocation is not currently being performed by running vxtask list, kill the vxrelocd process, and run nohup vxrelocd root user1 user2 ... &
Miscellaneous
Adding a disk to a disk group:
vxdiskadd disk_name
Creating a subdisk:
vxmake [-g groupname] sd subdisk diskname,offset,length
Creating a plex:
vxmake [-g groupname] plex plex sd=subdisk1[,subdisk2,...]
Creating a volume with vxmake:
vxmake [-g groupname] -U fsgen vol volume plex=plex1[,plex2,...]
Note: use gen instead of fsgen if you are creating a raw file system for RDBMS usage. fsgen is appropriate for general file system usage. More information on fsgen vs. gen.
After creating the volume, initialize the volume with vxvol start volume. If applicable, create the file system with newfs, create the mount point, and mount the volume as a file system.
Associating subdisks with plexes:
vxsd assoc plex subdisk1 [subdisk2 subdisk3 ...]
Displaying free disk space in a diskgroup:
vxdg [-g groupname] free
Dissociating subdisks from plexes:
vxsd dis subdisk
Dissociating subdisks from plexes, removing subdisk from VxVM:
vxsd -o rm dis subdisk
Dissociating and removing plexes and all associated subdisks:
vxplex -o rm dis plex
Removing a disk from a disk group:
vxdg [-g groupname] rmdisk diskname
Renaming a disk:
vxedit rename old_diskname new_diskname
Removing a volume (vxassist):
vxassist remove volume volume
Removing a volume (vxedit):
vxedit [-r] [-f] rm volume
-r -- recursive removal
-f -- force removal; needed if volume is enabled
Moving hot-relocated subdisks back to their original disk with vxunreloc:
/usr/lib/vxvm/bin/vxunreloc [-g groupname]original_disk
"vxvm:vxconfigd: ERROR: enable failed: Error in disk group configuration copies
Disk group has no valid configuration copies; transactions are disabled."
When receiving this error during system boot and when running vxinstall, follow the steps detailed in http://www.eng.auburn.edu/pub/mail-lists/veritas-users.May99/msg00048.html
In my case, the rootdg configuration was apparently corrupted. After issuing touch /etc/vx/reconfig.d/state.d/install-db and rebooting the machine, I was able to run vxinstall.
vxvm:vxdg: ERROR: Disk group disk_group: import failed: Disk group has no valid configuration copies"
This error can occur when attempting to import a disk group that was configured using a later version of VxVM. In this case, the disk group was configured with VxVM 3.2, but VxVM 3.1.1 was installed.
# pkginfo -l VRTSvxvm
PKGINST: VRTSvxvm
NAME: VERITAS Volume Manager, Binaries
CATEGORY: system
ARCH: sparc
VERSION: 3.1.1,REV=01.30.2001.22.21
Upgrading to at least the same version of VxVM used to configure the disk group will allow the disk group to be imported.
"ld.so.1: vxconfigd: fatal: libdevid.so.1: open failed: No such file or directory"
With Solaris 8 and VxVM 3.2, the shared library libdevid.so.1 does not get copied to /etc/vx/slib after installing Veritas. If you do not manually copy this shared library to /etc/vx/slib, your system will not boot. Follow these steps to make your system bootable:
1. Boot off a CD-ROM.
2. Mount your root and usr file systems.
3. Copy /usr/lib/libdevid.so.1 to /etc/vx/slib
4. Unmount your root and usr file systems and reboot.
More information:
http://marc.theaimsgroup.com/?l=veritas-vx&m=102636855529467&w=2
Clearing device locks
To clear a device lock, use the vxdisk clearimport command:
vxdisk clearimport devicename
ex. vxdisk clearimport c0t1d0
Using a Sun StorEdge A5000 disk array with Veritas
Make sure the array(s) are recognized by the operating system.
# luxadm probe
Found Enclosure(s):
SENA Name:a1 Node WWN:50800200000276e0
Logical Path:/dev/es/ses2
Logical Path:/dev/es/ses7
SENA Name:a2 Node WWN:5080020000028020
Logical Path:/dev/es/ses3
Logical Path:/dev/es/ses6
SENA Name:a0 Node WWN:5080020000026f38
Logical Path:/dev/es/ses4
Logical Path:/dev/es/ses5
SENA Name:a3 Node WWN:5080020000027060
Logical Path:/dev/es/ses8
Logical Path:/dev/es/ses9
Run Veritas' device discovery program.
# vxdctl enable
Determining maximum size of a volume
vxassist [ -g diskgroup ] maxsize layout=layout [attributes]
Example:
vxassist -g datadg maxsize layout=concat
layout may be concat, mirror, raid5, mirror-stripe, or stripe-mirror.
Veritas disk requirements
Disks managed by VxVM must have (1) two free partitions and (2) 2048 sectors of free space. The prtvtoc command displays how many sectors are in a disk cylinder:
# prtvtoc /dev/rdsk/c0t86d0s2
* /dev/rdsk/c0t86d0s2 partition map
*
* Dimensions:
* 512 bytes/sector
* 133 sectors/track
* 27 tracks/cylinder
* 3591 sectors/cylinder
* 4926 cylinders
* 4924 accessible cylinders
In this example, leave at least 1 cylinder free in your disk layout to allow for VxVM. If the disk is a boot disk, VxVM can shrink the swap partition to create space for VxVM's configuration data, but two free slices are essential for encapsulation.
Creating a volume with vxassist
ex.
# vxassist -g datadg maxsize
Maximum volume size: 35356672 (17264Mb)
# vxassist -g datadg make volume 35356672
Create the vxfs file system:
# mkfs -F vxfs /dev/vx/rdsk/datadg/db_backups
version 4 layout
35356672 sectors, 17678336 blocks of size 1024, log size 16384 blocks
unlimited inodes, largefiles not supported
17678336 data blocks, 17657432 free data blocks
540 allocation units of 32768 blocks, 32768 data blocks
last allocation unit has 16384 data blocks
Create the mount point:
# mkdir /db_backups
Mount the vxfs file system:
# mount -F vxfs /dev/vx/dsk/datadg/db_backups /db_backups
Add an /etc/vfstab entry to mount the file system after a reboot.
Replacing a failed disk
After replacing a failed disk in a SENA, make sure to run vxdctl enable for device discovery. Otherwise, you may encounter vxdmpadm errors:
Initialization of disk device c1t74d0 failed.
Error: vxvm:vxdmpadm: ERROR: Error in ioctl/open
vxdmpadm: No such file or directory
vxvm:vxdmpadm: ERROR: Invalid da_name
vxvm:vxdmpadm: ERROR: Invalid da_name
vxdisksetup: c1t74d0: Device address must be of the form cCtTdD or mcCtTdD where
C = host bus adapter controller number
T = target device controller number, if used
D = logical unit (disk) number within target device controller
# vxdisk list c1t74d0s2
Device: c1t74d0s2
devicetag: c1t74d0
type: sliced
flags: online error private autoconfig
errno: Device path not valid
Multipathing information:
numpaths: 2
c1t74d0s2 state=disabled
c5t74d0s2 state=disabled
When replacing a failed internal disk on a Sun E450 running Solaris 8, I had to spin the disk down using ssaadm stop /dev/rdsk/cxtxdxs2as the vxdiskadm's "Disable (offline) a disk device" did not seem to spin the disk down. If you are using a Sun system with FC-AL devices, you will want to use the luxadm command.
After replacing the disk, I enabled device discovery with vxdctl enable and un-relocated the failed subdisks back to this disk using /usr/lib/vxvm/bin/vxunreloc -g disk_group replaced_disk.
Adding additional users to VxVM electronic mail notifications
By default, VxVM sends electronic mail to the root user when failures are detected and hot-relocation is being performed. To notify additional users,
1. Edit /etc/init.d/vxvm-recover
2. Change the line containing vxrelocd root & to vxrelocd root user1 user2 ... &
This will preserve the change across system reboot.
3. To have the change take effect immediately, make sure that hot-relocation is not currently being performed by running vxtask list, kill the vxrelocd process, and run nohup vxrelocd root user1 user2 ... &
Miscellaneous
Adding a disk to a disk group:
vxdiskadd disk_name
Creating a subdisk:
vxmake [-g groupname] sd subdisk diskname,offset,length
Creating a plex:
vxmake [-g groupname] plex plex sd=subdisk1[,subdisk2,...]
Creating a volume with vxmake:
vxmake [-g groupname] -U fsgen vol volume plex=plex1[,plex2,...]
Note: use gen instead of fsgen if you are creating a raw file system for RDBMS usage. fsgen is appropriate for general file system usage. More information on fsgen vs. gen.
After creating the volume, initialize the volume with vxvol start volume. If applicable, create the file system with newfs, create the mount point, and mount the volume as a file system.
Associating subdisks with plexes:
vxsd assoc plex subdisk1 [subdisk2 subdisk3 ...]
Displaying free disk space in a diskgroup:
vxdg [-g groupname] free
Dissociating subdisks from plexes:
vxsd dis subdisk
Dissociating subdisks from plexes, removing subdisk from VxVM:
vxsd -o rm dis subdisk
Dissociating and removing plexes and all associated subdisks:
vxplex -o rm dis plex
Removing a disk from a disk group:
vxdg [-g groupname] rmdisk diskname
Renaming a disk:
vxedit rename old_diskname new_diskname
Removing a volume (vxassist):
vxassist remove volume volume
Removing a volume (vxedit):
vxedit [-r] [-f] rm volume
-r -- recursive removal
-f -- force removal; needed if volume is enabled
Moving hot-relocated subdisks back to their original disk with vxunreloc:
/usr/lib/vxvm/bin/vxunreloc [-g groupname]original_disk
Veritas licenses
The following information pertains to Veritas Volume Manager 3.2 for Solaris.
Location of license keys:
/etc/vx/elm
The key is the fourth line of the license file, below:
!
# DO NOT EDIT/COPY/MOVE/TOUCH THIS FILE!
# DOING SO WILL INVALIDATE THE KEY!
Check validity of license keys:
vxliccheck -pv
vrts:vxliccheck: INFO: License 95 valid
vrts:vxliccheck: INFO: License 96 valid
vrts:vxliccheck: INFO: License 98 valid
Print license details:
vxlicense -p
Create a license key file:
vxlicense -c
Location of license keys:
/etc/vx/elm
The key is the fourth line of the license file, below:
!
# DO NOT EDIT/COPY/MOVE/TOUCH THIS FILE!
# DOING SO WILL INVALIDATE THE KEY!
Check validity of license keys:
vxliccheck -pv
vrts:vxliccheck: INFO: License 95 valid
vrts:vxliccheck: INFO: License 96 valid
vrts:vxliccheck: INFO: License 98 valid
Print license details:
vxlicense -p
Create a license key file:
vxlicense -c
Unencapsulating a root disk
If your system partitions (/, swap, /usr, /var) are located on more than one physical disk, you will have to manually "unencapsulate" your root disk instead of using Veritas' vxunroot command below.
1. Modify /etc/vfstab to reference the cxtxdxsx devices instead of the VxVM devices.
2. Comment out the lines in /etc/system between:
* vxvm_START (do not remove)
* vxvm_END (do not remove)
3. Run the following command to prevent VxVM from starting up after reboot:
touch /etc/vx/reconfig.d/state.d/install-db
4. Reboot the system. After the reboot, you may uninstall VxVM if needed.
System partitions on boot disk
The Veritas vxunroot command is used to unencapsulate a root disk that contains all your system partitions. However, if the root disk is mirrored, you have to remove the mirror plexes.
Example:
# /etc/vx/bin/vxunroot
This operation will convert the following file systems from
volumes to regular partitions: root swap usr var opt home
ERROR: There are 2 plexes associated with volume rootvol
The vxunroot operation cannot proceed.
Listing of all volumes in rootdg:
# vxprint -v -g rootdg
TY NAME ASSOC KSTATE LENGTH PLOFFS STATE TUTIL0 PUTIL0
v opt gen ENABLED 4198392 - ACTIVE - -
v rootvol root ENABLED 1050776 - ACTIVE - -
v swapvol swap ENABLED 4198392 - ACTIVE - -
v usr gen ENABLED 4198392 - ACTIVE - -
v var gen ENABLED 4198392 - ACTIVE - -
Here we see that rootdg contains volumes opt, rootvol, swapvol, usr, and var. Let's see if the volumes consist of more than one plex.
# vxprint opt rootvol swapvol usr var
Disk group: rootdg
TY NAME ASSOC KSTATE LENGTH PLOFFS STATE TUTIL0 PUTIL0
v opt gen ENABLED 4198392 - ACTIVE - -
pl opt-01 opt ENABLED 4198392 - ACTIVE - -
sd rootdisk-04 opt-01 ENABLED 4198392 0 - - -
pl opt-02 opt ENABLED 4198392 - ACTIVE - -
sd rootdisk-mirror-01 opt-02 ENABLED 4198392 0 - - -
v rootvol root ENABLED 1050776 - ACTIVE - -
pl rootvol-01 rootvol ENABLED 1050776 - ACTIVE - -
sd rootdisk-B0 rootvol-01 ENABLED 1 0 - - Block0
pl rootvol-02 rootvol ENABLED 1050776 - ACTIVE - -
sd rootdisk-02 rootvol-01 ENABLED 1050775 1 - - -
v swapvol swap ENABLED 4198392 - ACTIVE - -
pl swapvol-01 swapvol ENABLED 4198392 - ACTIVE - -
sd rootdisk-01 swapvol-01 ENABLED 4198392 0 - - -
pl swapvol-02 swapvol ENABLED 4198392 - ACTIVE - -
sd rootdisk-mirror-03 swapvol-02 ENABLED 4198392 0 - - -
v usr gen ENABLED 4198392 - ACTIVE - -
pl usr-01 usr ENABLED 4198392 - ACTIVE - -
sd rootdisk-03 usr-01 ENABLED 4198392 0 - - -
pl usr-02 usr ENABLED 4198392 - ACTIVE - -
sd rootdisk-mirror-04 usr-02 ENABLED 4198392 0 - - -
v var gen ENABLED 4198392 - ACTIVE - -
pl var-01 var ENABLED 4198392 - ACTIVE - -
sd rootdisk-05 var-01 ENABLED 4198392 0 - - -
pl var-02 var ENABLED 4198392 - ACTIVE - -
sd rootdisk-mirror-05 var-02 ENABLED 4198392 0 - - -
VM disk rootdisk-mirror contains mirror plexes for volumes opt,rootvol, swapvol, usr, and var. We have to remove the plexes before proceeding with vxunroot.
# vxplex -o rm dis opt-02 rootvol-02 swapvol-02 usr-02 var-02
# vxprint opt rootvol swapvol usr var
Disk group: rootdg
TY NAME ASSOC KSTATE LENGTH PLOFFS STATE TUTIL0 PUTIL0
v opt gen ENABLED 4198392 - ACTIVE - -
pl opt-01 opt ENABLED 4198392 - ACTIVE - -
sd rootdisk-04 opt-01 ENABLED 4198392 0 - - -
v rootvol root ENABLED 1050776 - ACTIVE - -
pl rootvol-01 rootvol ENABLED 1050776 - ACTIVE - -
sd rootdisk-B0 rootvol-01 ENABLED 1 0 - - Block0
sd rootdisk-02 rootvol-01 ENABLED 1050775 1 - - -
v swapvol swap ENABLED 4198392 - ACTIVE - -
pl swapvol-01 swapvol ENABLED 4198392 - ACTIVE - -
sd rootdisk-01 swapvol-01 ENABLED 4198392 0 - - -
v usr gen ENABLED 4198392 - ACTIVE - -
pl usr-01 usr ENABLED 4198392 - ACTIVE - -
sd rootdisk-03 usr-01 ENABLED 4198392 0 - - -
v var gen ENABLED 4198392 - ACTIVE - -
pl var-01 var ENABLED 4198392 - ACTIVE - -
sd rootdisk-05 var-01 ENABLED 4198392 0 - - -
# /etc/vx/bin/vxunroot
This operation will convert the following file systems from
volumes to regular partitions: root swap usr var opt home
Replace volume rootvol with c0t0d0s0.
This operation will require a system reboot. If you choose to
continue with this operation, system configuration will be updated
to discontinue use of the volume manager for your root and swap
devices.
Do you wish to do this now [y,n,q,?] (default: y)
After a reboot, the root disk will be unencapsulated.
1. Modify /etc/vfstab to reference the cxtxdxsx devices instead of the VxVM devices.
2. Comment out the lines in /etc/system between:
* vxvm_START (do not remove)
* vxvm_END (do not remove)
3. Run the following command to prevent VxVM from starting up after reboot:
touch /etc/vx/reconfig.d/state.d/install-db
4. Reboot the system. After the reboot, you may uninstall VxVM if needed.
System partitions on boot disk
The Veritas vxunroot command is used to unencapsulate a root disk that contains all your system partitions. However, if the root disk is mirrored, you have to remove the mirror plexes.
Example:
# /etc/vx/bin/vxunroot
This operation will convert the following file systems from
volumes to regular partitions: root swap usr var opt home
ERROR: There are 2 plexes associated with volume rootvol
The vxunroot operation cannot proceed.
Listing of all volumes in rootdg:
# vxprint -v -g rootdg
TY NAME ASSOC KSTATE LENGTH PLOFFS STATE TUTIL0 PUTIL0
v opt gen ENABLED 4198392 - ACTIVE - -
v rootvol root ENABLED 1050776 - ACTIVE - -
v swapvol swap ENABLED 4198392 - ACTIVE - -
v usr gen ENABLED 4198392 - ACTIVE - -
v var gen ENABLED 4198392 - ACTIVE - -
Here we see that rootdg contains volumes opt, rootvol, swapvol, usr, and var. Let's see if the volumes consist of more than one plex.
# vxprint opt rootvol swapvol usr var
Disk group: rootdg
TY NAME ASSOC KSTATE LENGTH PLOFFS STATE TUTIL0 PUTIL0
v opt gen ENABLED 4198392 - ACTIVE - -
pl opt-01 opt ENABLED 4198392 - ACTIVE - -
sd rootdisk-04 opt-01 ENABLED 4198392 0 - - -
pl opt-02 opt ENABLED 4198392 - ACTIVE - -
sd rootdisk-mirror-01 opt-02 ENABLED 4198392 0 - - -
v rootvol root ENABLED 1050776 - ACTIVE - -
pl rootvol-01 rootvol ENABLED 1050776 - ACTIVE - -
sd rootdisk-B0 rootvol-01 ENABLED 1 0 - - Block0
pl rootvol-02 rootvol ENABLED 1050776 - ACTIVE - -
sd rootdisk-02 rootvol-01 ENABLED 1050775 1 - - -
v swapvol swap ENABLED 4198392 - ACTIVE - -
pl swapvol-01 swapvol ENABLED 4198392 - ACTIVE - -
sd rootdisk-01 swapvol-01 ENABLED 4198392 0 - - -
pl swapvol-02 swapvol ENABLED 4198392 - ACTIVE - -
sd rootdisk-mirror-03 swapvol-02 ENABLED 4198392 0 - - -
v usr gen ENABLED 4198392 - ACTIVE - -
pl usr-01 usr ENABLED 4198392 - ACTIVE - -
sd rootdisk-03 usr-01 ENABLED 4198392 0 - - -
pl usr-02 usr ENABLED 4198392 - ACTIVE - -
sd rootdisk-mirror-04 usr-02 ENABLED 4198392 0 - - -
v var gen ENABLED 4198392 - ACTIVE - -
pl var-01 var ENABLED 4198392 - ACTIVE - -
sd rootdisk-05 var-01 ENABLED 4198392 0 - - -
pl var-02 var ENABLED 4198392 - ACTIVE - -
sd rootdisk-mirror-05 var-02 ENABLED 4198392 0 - - -
VM disk rootdisk-mirror contains mirror plexes for volumes opt,rootvol, swapvol, usr, and var. We have to remove the plexes before proceeding with vxunroot.
# vxplex -o rm dis opt-02 rootvol-02 swapvol-02 usr-02 var-02
# vxprint opt rootvol swapvol usr var
Disk group: rootdg
TY NAME ASSOC KSTATE LENGTH PLOFFS STATE TUTIL0 PUTIL0
v opt gen ENABLED 4198392 - ACTIVE - -
pl opt-01 opt ENABLED 4198392 - ACTIVE - -
sd rootdisk-04 opt-01 ENABLED 4198392 0 - - -
v rootvol root ENABLED 1050776 - ACTIVE - -
pl rootvol-01 rootvol ENABLED 1050776 - ACTIVE - -
sd rootdisk-B0 rootvol-01 ENABLED 1 0 - - Block0
sd rootdisk-02 rootvol-01 ENABLED 1050775 1 - - -
v swapvol swap ENABLED 4198392 - ACTIVE - -
pl swapvol-01 swapvol ENABLED 4198392 - ACTIVE - -
sd rootdisk-01 swapvol-01 ENABLED 4198392 0 - - -
v usr gen ENABLED 4198392 - ACTIVE - -
pl usr-01 usr ENABLED 4198392 - ACTIVE - -
sd rootdisk-03 usr-01 ENABLED 4198392 0 - - -
v var gen ENABLED 4198392 - ACTIVE - -
pl var-01 var ENABLED 4198392 - ACTIVE - -
sd rootdisk-05 var-01 ENABLED 4198392 0 - - -
# /etc/vx/bin/vxunroot
This operation will convert the following file systems from
volumes to regular partitions: root swap usr var opt home
Replace volume rootvol with c0t0d0s0.
This operation will require a system reboot. If you choose to
continue with this operation, system configuration will be updated
to discontinue use of the volume manager for your root and swap
devices.
Do you wish to do this now [y,n,q,?] (default: y)
After a reboot, the root disk will be unencapsulated.
Resizing a file system
In this example, I will resize a UFS file system under VxVM control from 3GB to 4GB using vxresize.
Current capacity:
# df -k /dbfiles03
Filesystem kbytes used avail capacity Mounted on
/dev/vx/dsk/dg20/dbvol03
3079710 2709166 308950 90% /dbfiles03
File system type:
# mount -v | grep /dbfiles03
/dev/vx/dsk/dg20/dbvol03 on /dbfiles03 type ufs read/write/setuid/intr/largefiles/onerror=panic/dev=3d1349e on Sun Aug 3 16:21:54 2003
Volume information:
# vxprint dbvol03
Disk group: dg20
TY NAME ASSOC KSTATE LENGTH PLOFFS STATE TUTIL0 PUTIL0
v dbvol03 fsgen ENABLED 6291456 - ACTIVE - -
pl dbvol03-01 dbvol03 ENABLED 6298619 - ACTIVE - -
sd dg2007-03 dbvol03-01 ENABLED 3149307 0 - - -
sd dg2006-03 dbvol03-01 ENABLED 3149307 0 - - -
Plex information:
# vxprint -l dbvol03-01
Disk group: dg20
Plex: dbvol03-01
info: len=6298619 contiglen=6298491
type: layout=STRIPE columns=2 width=128
state: state=ACTIVE kernel=ENABLED io=read-write
assoc: vol=dbvol03 sd=dg2007-03,dg2006-03
flags: busy complete
Increasing the volume to 4GB using vxresize:
# vxresize dbvol03 4g
/dev/vx/rdsk/dg20/dbvol03: 8388608 sectors in 4096 cylinders of 32 tracks, 64 sectors
4096.0MB in 88 cyl groups (47 c/g, 47.00MB/g, 7872 i/g)
super-block backups (for fsck -F ufs -o b=#) at:
32, 96352, 192672, 288992, 385312, 481632, 577952, 674272, 770592, 866912,
963232, 1059552, 1155872, 1252192, 1348512, 1444832, 1541152, 1637472,
1733792, 1830112, 1926432, 2022752, 2119072, 2215392, 2311712, 2408032,
2504352, 2600672, 2696992, 2793312, 2889632, 2985952, 3080224, 3176544,
3272864, 3369184, 3465504, 3561824, 3658144, 3754464, 3850784, 3947104,
4043424, 4139744, 4236064, 4332384, 4428704, 4525024, 4621344, 4717664,
4813984, 4910304, 5006624, 5102944, 5199264, 5295584, 5391904, 5488224,
5584544, 5680864, 5777184, 5873504, 5969824, 6066144, 6160416, 6256736,
6353056, 6449376, 6545696, 6642016, 6738336, 6834656, 6930976, 7027296,
7123616, 7219936, 7316256, 7412576, 7508896, 7605216, 7701536, 7797856,
7894176, 7990496, 8086816, 8183136, 8279456, 8375776,
New capacity:
# df -k /dbfiles03
Filesystem kbytes used avail capacity Mounted on
/dev/vx/dsk/dg20/dbvol03
4106286 2709166 1335526 67% /dbfiles03
New volume information (two new subdisks):
# vxprint dbvol03
Disk group: dg20
TY NAME ASSOC KSTATE LENGTH PLOFFS STATE TUTIL0 PUTIL0
v dbvol03 fsgen ENABLED 8388608 - ACTIVE - -
pl dbvol03-01 dbvol03 ENABLED 8395767 - ACTIVE - -
sd dg2007-03 dbvol03-01 ENABLED 3149307 0 - - -
sd dg2007-05 dbvol03-01 ENABLED 1048572 3149307 - - -
sd dg2006-03 dbvol03-01 ENABLED 3149307 0 - - -
sd dg2006-05 dbvol03-01 ENABLED 1048572 3149307 - - -
New plex information:
# vxprint -l dbvol03-01
Disk group: dg20
Plex: dbvol03-01
info: len=8395767 contiglen=8395639
type: layout=STRIPE columns=2 width=128
state: state=ACTIVE kernel=ENABLED io=read-write
assoc: vol=dbvol03 sd=dg2007-03,dg2007-05,dg2006-03,dg2006-05
flags: busy complete
Current capacity:
# df -k /dbfiles03
Filesystem kbytes used avail capacity Mounted on
/dev/vx/dsk/dg20/dbvol03
3079710 2709166 308950 90% /dbfiles03
File system type:
# mount -v | grep /dbfiles03
/dev/vx/dsk/dg20/dbvol03 on /dbfiles03 type ufs read/write/setuid/intr/largefiles/onerror=panic/dev=3d1349e on Sun Aug 3 16:21:54 2003
Volume information:
# vxprint dbvol03
Disk group: dg20
TY NAME ASSOC KSTATE LENGTH PLOFFS STATE TUTIL0 PUTIL0
v dbvol03 fsgen ENABLED 6291456 - ACTIVE - -
pl dbvol03-01 dbvol03 ENABLED 6298619 - ACTIVE - -
sd dg2007-03 dbvol03-01 ENABLED 3149307 0 - - -
sd dg2006-03 dbvol03-01 ENABLED 3149307 0 - - -
Plex information:
# vxprint -l dbvol03-01
Disk group: dg20
Plex: dbvol03-01
info: len=6298619 contiglen=6298491
type: layout=STRIPE columns=2 width=128
state: state=ACTIVE kernel=ENABLED io=read-write
assoc: vol=dbvol03 sd=dg2007-03,dg2006-03
flags: busy complete
Increasing the volume to 4GB using vxresize:
# vxresize dbvol03 4g
/dev/vx/rdsk/dg20/dbvol03: 8388608 sectors in 4096 cylinders of 32 tracks, 64 sectors
4096.0MB in 88 cyl groups (47 c/g, 47.00MB/g, 7872 i/g)
super-block backups (for fsck -F ufs -o b=#) at:
32, 96352, 192672, 288992, 385312, 481632, 577952, 674272, 770592, 866912,
963232, 1059552, 1155872, 1252192, 1348512, 1444832, 1541152, 1637472,
1733792, 1830112, 1926432, 2022752, 2119072, 2215392, 2311712, 2408032,
2504352, 2600672, 2696992, 2793312, 2889632, 2985952, 3080224, 3176544,
3272864, 3369184, 3465504, 3561824, 3658144, 3754464, 3850784, 3947104,
4043424, 4139744, 4236064, 4332384, 4428704, 4525024, 4621344, 4717664,
4813984, 4910304, 5006624, 5102944, 5199264, 5295584, 5391904, 5488224,
5584544, 5680864, 5777184, 5873504, 5969824, 6066144, 6160416, 6256736,
6353056, 6449376, 6545696, 6642016, 6738336, 6834656, 6930976, 7027296,
7123616, 7219936, 7316256, 7412576, 7508896, 7605216, 7701536, 7797856,
7894176, 7990496, 8086816, 8183136, 8279456, 8375776,
New capacity:
# df -k /dbfiles03
Filesystem kbytes used avail capacity Mounted on
/dev/vx/dsk/dg20/dbvol03
4106286 2709166 1335526 67% /dbfiles03
New volume information (two new subdisks):
# vxprint dbvol03
Disk group: dg20
TY NAME ASSOC KSTATE LENGTH PLOFFS STATE TUTIL0 PUTIL0
v dbvol03 fsgen ENABLED 8388608 - ACTIVE - -
pl dbvol03-01 dbvol03 ENABLED 8395767 - ACTIVE - -
sd dg2007-03 dbvol03-01 ENABLED 3149307 0 - - -
sd dg2007-05 dbvol03-01 ENABLED 1048572 3149307 - - -
sd dg2006-03 dbvol03-01 ENABLED 3149307 0 - - -
sd dg2006-05 dbvol03-01 ENABLED 1048572 3149307 - - -
New plex information:
# vxprint -l dbvol03-01
Disk group: dg20
Plex: dbvol03-01
info: len=8395767 contiglen=8395639
type: layout=STRIPE columns=2 width=128
state: state=ACTIVE kernel=ENABLED io=read-write
assoc: vol=dbvol03 sd=dg2007-03,dg2007-05,dg2006-03,dg2006-05
flags: busy complete
Veritas Volume manager
Creating a volume with vxmake
In this example, I create a 26 GB concatenated volume named EZTK-NEW using disks in disk group dg15. The volume consists of 3 plexes (3 copies of the data). Each plex is composed of two 13 GB subdisks.
1. Identify disks in disk group dg15 that have enough free space to create a 13 GB subdisk.
# vxdg -g dg15 free
DISK DEVICE TAG OFFSET LENGTH FLAGS
S-f0 c1t0d0s2 c1t0d0 35302304 61256 -
S-f1 c1t1d0s2 c1t1d0 35302304 61256 -
S-f2 c1t2d0s2 c1t2d0 35302304 61256 -
S-f3 c1t3d0s2 c1t3d0 35302304 61256 -
S-f4 c1t4d0s2 c1t4d0 35302304 61256 -
S-f6 c1t6d0s2 c1t6d0 35302304 61256 -
S-f9 c1t9d0s2 c1t9d0 17062152 18301408 -
S-f10 c1t10d0s2 c1t10d0 20973112 14385736 -
The LENGTH column displays the number of free sectors on the disk (each sector is 512 bytes). Although not displayed here, disks b1-r2, b1-r9, b1-f4, b1-f6, b2-f4, and b2-r2 have enough free space to create 13 GB subdisks.
2. Create the subdisks.
Syntax:
# vxmake sd subdisk diskname,offset,length
Plex one:
# vxmake -g dg15 sd b1-r2-01 b1-r2,0,13g
# vxmake -g dg15 sd b1-r9-01 b1-r9,0,13g
Plex two:
# vxmake -g dg15 sd b1-f4-01 b1-f4,6582664,13g
# vxmake -g dg15 sd b1-f6-01 b1-f6,0,13g
Plex three:
# vxmake -g dg15 sd b2-f4-01 b2-f4,6582664,13g
# vxmake -g dg15 sd b2-r2-01 b2-r2,0,13g
3. Create the three plexes and associate the subdisks with them.
Syntax:
# vxmake plex plex sd=subdisk1[,subdisk2,...]
Plex one named EZTK-P01:
# vxmake -g dg15 plex EZTK-NEW-P01 sd=b1-r2-01,b1-r9-01
Plex two named EZTK-P02:
# vxmake -g dg15 plex EZTK-NEW-P02 sd=b1-f4-01,b1-f6-01
Plex three named EZTK-P03:
# vxmake -g dg15 plex EZTK-NEW-P03 sd=b2-f4-01,b2-r2-01
4. Create the volume consisting of the three plexes.
Creating volume EZTK-NEW composed of plexes EZTK-P01, EZTK-P02, and EZTK-P03:
# vxmake -g dg15 -U gen vol EZTK-NEW plex=EZTK-NEW-P01,EZTK-NEW-P02,EZTK-NEW-P03
5. Initialize the volume.
# vxvol start EZTK-NEW
The volume has been created. Before you are able to mount this volume as a file system, you will have to create a file system (UFS or vxfs) using newfs.
In this example, I create a 26 GB concatenated volume named EZTK-NEW using disks in disk group dg15. The volume consists of 3 plexes (3 copies of the data). Each plex is composed of two 13 GB subdisks.
1. Identify disks in disk group dg15 that have enough free space to create a 13 GB subdisk.
# vxdg -g dg15 free
DISK DEVICE TAG OFFSET LENGTH FLAGS
S-f0 c1t0d0s2 c1t0d0 35302304 61256 -
S-f1 c1t1d0s2 c1t1d0 35302304 61256 -
S-f2 c1t2d0s2 c1t2d0 35302304 61256 -
S-f3 c1t3d0s2 c1t3d0 35302304 61256 -
S-f4 c1t4d0s2 c1t4d0 35302304 61256 -
S-f6 c1t6d0s2 c1t6d0 35302304 61256 -
S-f9 c1t9d0s2 c1t9d0 17062152 18301408 -
S-f10 c1t10d0s2 c1t10d0 20973112 14385736 -
The LENGTH column displays the number of free sectors on the disk (each sector is 512 bytes). Although not displayed here, disks b1-r2, b1-r9, b1-f4, b1-f6, b2-f4, and b2-r2 have enough free space to create 13 GB subdisks.
2. Create the subdisks.
Syntax:
# vxmake sd subdisk diskname,offset,length
Plex one:
# vxmake -g dg15 sd b1-r2-01 b1-r2,0,13g
# vxmake -g dg15 sd b1-r9-01 b1-r9,0,13g
Plex two:
# vxmake -g dg15 sd b1-f4-01 b1-f4,6582664,13g
# vxmake -g dg15 sd b1-f6-01 b1-f6,0,13g
Plex three:
# vxmake -g dg15 sd b2-f4-01 b2-f4,6582664,13g
# vxmake -g dg15 sd b2-r2-01 b2-r2,0,13g
3. Create the three plexes and associate the subdisks with them.
Syntax:
# vxmake plex plex sd=subdisk1[,subdisk2,...]
Plex one named EZTK-P01:
# vxmake -g dg15 plex EZTK-NEW-P01 sd=b1-r2-01,b1-r9-01
Plex two named EZTK-P02:
# vxmake -g dg15 plex EZTK-NEW-P02 sd=b1-f4-01,b1-f6-01
Plex three named EZTK-P03:
# vxmake -g dg15 plex EZTK-NEW-P03 sd=b2-f4-01,b2-r2-01
4. Create the volume consisting of the three plexes.
Creating volume EZTK-NEW composed of plexes EZTK-P01, EZTK-P02, and EZTK-P03:
# vxmake -g dg15 -U gen vol EZTK-NEW plex=EZTK-NEW-P01,EZTK-NEW-P02,EZTK-NEW-P03
5. Initialize the volume.
# vxvol start EZTK-NEW
The volume has been created. Before you are able to mount this volume as a file system, you will have to create a file system (UFS or vxfs) using newfs.
Web traffic script
Web traffic script
The Web pages are hosted on a FreeBSD Web server. FreeBSD's date command makes determining yesterday's date easy. The script is run via the cron daemon every morning to email me the results.
#!/bin/sh
# Get yesterday's date in dd/mmm/yyyy format
YESTERDAYS_DATE=`date -v -1d +%d/%b/%Y`
# Get yesterday's access_log information in a temporary file
grep $YESTERDAYS_DATE /www/logs/access_log > /tmp/access_log
# For each Web page, count the number of hits
for i in `cd /www/htdocs/brandonhutchinson; ls *.html`
do
echo -e "`grep $i /tmp/access_log | sort -u | wc -l` $i" >> /tmp/web_traffic.txt
done
# Tally and print the total number of visitors; mail results
(echo -e "Total visitors: `awk '{ hits += $1 } END { print hits }' < /tmp/web_traffic.txt`\n" && sort -rn /tmp/web_traffic.txt) | mail -s "brandonhutchinson.com Web traffic" brandonhutchinson@hotmail.com
# Remove temporary files
rm /tmp/access_log
rm /tmp/web_traffic.txt
The Web pages are hosted on a FreeBSD Web server. FreeBSD's date command makes determining yesterday's date easy. The script is run via the cron daemon every morning to email me the results.
#!/bin/sh
# Get yesterday's date in dd/mmm/yyyy format
YESTERDAYS_DATE=`date -v -1d +%d/%b/%Y`
# Get yesterday's access_log information in a temporary file
grep $YESTERDAYS_DATE /www/logs/access_log > /tmp/access_log
# For each Web page, count the number of hits
for i in `cd /www/htdocs/brandonhutchinson; ls *.html`
do
echo -e "`grep $i /tmp/access_log | sort -u | wc -l` $i" >> /tmp/web_traffic.txt
done
# Tally and print the total number of visitors; mail results
(echo -e "Total visitors: `awk '{ hits += $1 } END { print hits }' < /tmp/web_traffic.txt`\n" && sort -rn /tmp/web_traffic.txt) | mail -s "brandonhutchinson.com Web traffic" brandonhutchinson@hotmail.com
# Remove temporary files
rm /tmp/access_log
rm /tmp/web_traffic.txt
Testing Fire wall rules
Testing firewall rules
Sometimes it is handy to check firewall rules without coordinating a test with the end user. For these tests, use the hping2 utility to "spoof" traffic coming from the source IP address(es) used in the firewall rules.
At the same time, monitor the internal and external network interfaces on the firewall to make sure traffic is reaching the firewall and allowed through the firewall. In order to do this, you must have root access on the firewall and on the machine running hping2.
Example firewall rule:
Permit source IP 192.168.1.1 to communicate with destination IP 10.0.0.1 over TCP port 1000.
To test the rule, issue the following hping2 command:
hping2 -a 192.168.1.1 10.0.0.1 -p 1000
At the same time, log into the firewall and run the following commands (example using a Solaris firewall with internal network interface hme0 and external network interface qfe0):
In window 1:
snoop -d hme0 host 192.168.1.1 port 1000
-- or --
tcpdump -i hme0 host 192.168.1.1 and port 1000
In window 2:
snoop -d qfe0 host 10.0.0.1 port 1000
-- or --
tcpdump -i qfe0 host 10.0.0.1 and port 1000
If you do not see any output in window 1, traffic is not reaching the firewall. A choke router or other packet-filtering device may not be allowing the traffic to reach the firewall.
If you see output in window 1 but not in window 2, traffic is not being allowed through the firewall. Check the firewall rulebase for any errors.
Finally I want say thanks to http://brandonhutchinson.com for helping me in critical situation
Sometimes it is handy to check firewall rules without coordinating a test with the end user. For these tests, use the hping2 utility to "spoof" traffic coming from the source IP address(es) used in the firewall rules.
At the same time, monitor the internal and external network interfaces on the firewall to make sure traffic is reaching the firewall and allowed through the firewall. In order to do this, you must have root access on the firewall and on the machine running hping2.
Example firewall rule:
Permit source IP 192.168.1.1 to communicate with destination IP 10.0.0.1 over TCP port 1000.
To test the rule, issue the following hping2 command:
hping2 -a 192.168.1.1 10.0.0.1 -p 1000
At the same time, log into the firewall and run the following commands (example using a Solaris firewall with internal network interface hme0 and external network interface qfe0):
In window 1:
snoop -d hme0 host 192.168.1.1 port 1000
-- or --
tcpdump -i hme0 host 192.168.1.1 and port 1000
In window 2:
snoop -d qfe0 host 10.0.0.1 port 1000
-- or --
tcpdump -i qfe0 host 10.0.0.1 and port 1000
If you do not see any output in window 1, traffic is not reaching the firewall. A choke router or other packet-filtering device may not be allowing the traffic to reach the firewall.
If you see output in window 1 but not in window 2, traffic is not being allowed through the firewall. Check the firewall rulebase for any errors.
Finally I want say thanks to http://brandonhutchinson.com for helping me in critical situation
Stand alone Iptables
Standalone iptables firewall
The following standalone iptables firewall is suited for a machine with one network interface that allows unlimited loopback traffic and outbound traffic, but does not run any services requiring incoming connection requests. Incoming ICMP ECHO REQUESTS ("pings") are allowed, while all incoming connection requests are silently dropped.
#!/bin/sh
# Kernel monitoring support
# More information:
# /usr/src/linux-`uname -r`/Documentation/networking/ip-sysctl.txt
# http://www.linuxgazette.com/book/view/1645
# http://www.spirit.com/Network/net0300.html
# Drop ICMP echo-request messages sent to broadcast or multicast addresses
echo 1 > /proc/sys/net/ipv4/icmp_echo_ignore_broadcasts
# Drop source routed packets
echo 0 > /proc/sys/net/ipv4/conf/all/accept_source_route
# Enable TCP SYN cookie protection from SYN floods
echo 1 > /proc/sys/net/ipv4/tcp_syncookies
# Don't accept ICMP redirect messages
echo 0 > /proc/sys/net/ipv4/conf/all/accept_redirects
# Don't send ICMP redirect messages
echo 0 > /proc/sys/net/ipv4/conf/all/send_redirects
# Enable source address spoofing protection
echo 1 > /proc/sys/net/ipv4/conf/all/rp_filter
# Log packets with impossible source addresses
echo 1 > /proc/sys/net/ipv4/conf/all/log_martians
# Flush all chains
/sbin/iptables --flush
# Allow unlimited traffic on the loopback interface
/sbin/iptables -A INPUT -i lo -j ACCEPT
/sbin/iptables -A OUTPUT -o lo -j ACCEPT
# Set default policies
/sbin/iptables --policy INPUT DROP
/sbin/iptables --policy OUTPUT DROP
/sbin/iptables --policy FORWARD DROP
# Previously initiated and accepted exchanges bypass rule checking
# Allow unlimited outbound traffic
/sbin/iptables -A INPUT -m state --state ESTABLISHED,RELATED -j ACCEPT
/sbin/iptables -A OUTPUT -m state --state NEW,ESTABLISHED,RELATED -j ACCEPT
# Allow ICMP ECHO REQUESTS from anywhere
/sbin/iptables -A INPUT -p icmp --icmp-type echo-request -j ACCEPT
# Drop all other traffic
/sbin/iptables -A INPUT -j DROP
# Have these rules take effect when iptables is started
/sbin/service iptables save
The following standalone iptables firewall is suited for a machine with one network interface that allows unlimited loopback traffic and outbound traffic, but does not run any services requiring incoming connection requests. Incoming ICMP ECHO REQUESTS ("pings") are allowed, while all incoming connection requests are silently dropped.
#!/bin/sh
# Kernel monitoring support
# More information:
# /usr/src/linux-`uname -r`/Documentation/networking/ip-sysctl.txt
# http://www.linuxgazette.com/book/view/1645
# http://www.spirit.com/Network/net0300.html
# Drop ICMP echo-request messages sent to broadcast or multicast addresses
echo 1 > /proc/sys/net/ipv4/icmp_echo_ignore_broadcasts
# Drop source routed packets
echo 0 > /proc/sys/net/ipv4/conf/all/accept_source_route
# Enable TCP SYN cookie protection from SYN floods
echo 1 > /proc/sys/net/ipv4/tcp_syncookies
# Don't accept ICMP redirect messages
echo 0 > /proc/sys/net/ipv4/conf/all/accept_redirects
# Don't send ICMP redirect messages
echo 0 > /proc/sys/net/ipv4/conf/all/send_redirects
# Enable source address spoofing protection
echo 1 > /proc/sys/net/ipv4/conf/all/rp_filter
# Log packets with impossible source addresses
echo 1 > /proc/sys/net/ipv4/conf/all/log_martians
# Flush all chains
/sbin/iptables --flush
# Allow unlimited traffic on the loopback interface
/sbin/iptables -A INPUT -i lo -j ACCEPT
/sbin/iptables -A OUTPUT -o lo -j ACCEPT
# Set default policies
/sbin/iptables --policy INPUT DROP
/sbin/iptables --policy OUTPUT DROP
/sbin/iptables --policy FORWARD DROP
# Previously initiated and accepted exchanges bypass rule checking
# Allow unlimited outbound traffic
/sbin/iptables -A INPUT -m state --state ESTABLISHED,RELATED -j ACCEPT
/sbin/iptables -A OUTPUT -m state --state NEW,ESTABLISHED,RELATED -j ACCEPT
# Allow ICMP ECHO REQUESTS from anywhere
/sbin/iptables -A INPUT -p icmp --icmp-type echo-request -j ACCEPT
# Drop all other traffic
/sbin/iptables -A INPUT -j DROP
# Have these rules take effect when iptables is started
/sbin/service iptables save
Squid iptables
Squid iptables firewall
The following iptables firewall is suited for a dual-homed Squid proxy server. ssh (TCP port 22), squid (TCP port 3128), and ICMP ECHO requests are allowed on the internal (LAN) interface.
Squid is configured to proxy ftp, http, https, and AOL Instant Messenger traffic. In addition, the server is running a caching/forwarding name server and time server and therefore requires therefore requires outgoing UDP port 123 (ntp) and TCP/UDP port 53 (dns).
#!/bin/sh
LAN="eth1"
INTERNET="eth0"
IPTABLES="/sbin/iptables"
# Kernel monitoring support
# More information:
# /usr/src/linux-`uname -r`/Documentation/networking/ip-sysctl.txt
# http://www.linuxgazette.com/book/view/1645
# http://www.spirit.com/Network/net0300.html
# Drop ICMP echo-request messages sent to broadcast or multicast addresses
echo 1 > /proc/sys/net/ipv4/icmp_echo_ignore_broadcasts
# Drop source routed packets
echo 0 > /proc/sys/net/ipv4/conf/all/accept_source_route
# Enable TCP SYN cookie protection from SYN floods
echo 1 > /proc/sys/net/ipv4/tcp_syncookies
# Don't accept ICMP redirect messages
echo 0 > /proc/sys/net/ipv4/conf/all/accept_redirects
# Don't send ICMP redirect messages
echo 0 > /proc/sys/net/ipv4/conf/all/send_redirects
# Enable source address spoofing protection
echo 1 > /proc/sys/net/ipv4/conf/all/rp_filter
# Log packets with impossible source addresses
echo 1 > /proc/sys/net/ipv4/conf/all/log_martians
# Needed for FTP (specifically, to allow incoming ftp-data connections)
/sbin/modprobe ip_conntrack_ftp
# Flush all chains
$IPTABLES --flush
# Allow unlimited traffic on the loopback interface
$IPTABLES -A INPUT -i lo -j ACCEPT
$IPTABLES -A OUTPUT -o lo -j ACCEPT
# Set default policies
$IPTABLES --policy INPUT DROP
$IPTABLES --policy OUTPUT DROP
$IPTABLES --policy FORWARD DROP
# Previously initiated and accepted exchanges bypass rule checking
$IPTABLES -A INPUT -m state --state ESTABLISHED,RELATED -j ACCEPT
$IPTABLES -A OUTPUT -m state --state ESTABLISHED,RELATED -j ACCEPT
# Allow incoming port 22 (ssh) connections on LAN interface
$IPTABLES -A INPUT -i $LAN -p tcp --destination-port 22 -m state \
--state NEW -j ACCEPT
# Allow incoming port 3128 (squid) connections on LAN interface
$IPTABLES -A INPUT -i $LAN -p tcp --destination-port 3128 -m state \
--state NEW -j ACCEPT
# Allow ICMP ECHO REQUESTS on LAN interface
$IPTABLES -A INPUT -i $LAN -p icmp --icmp-type echo-request -j ACCEPT
# Allow DNS resolution
$IPTABLES -A OUTPUT -o $INTERNET -p udp --destination-port 53 -m state \
--state NEW -j ACCEPT
$IPTABLES -A OUTPUT -o $INTERNET -p tcp --destination-port 53 -m state \
--state NEW -j ACCEPT
# Allow ntp synchronization
$IPTABLES -A OUTPUT -o $LAN -p udp --destination-port 123 -m state \
--state NEW -j ACCEPT
# Allow ssh on LAN interface
$IPTABLES -A OUTPUT -o $LAN -p tcp --destination-port 22 -m state \
--state NEW -j ACCEPT
# Allow Squid to proxy ftp, http, https, and AIM traffic
$IPTABLES -A OUTPUT -o $INTERNET -p tcp --destination-port 21 -m state \
--state NEW -j ACCEPT
$IPTABLES -A OUTPUT -o $INTERNET -p tcp --destination-port 80 -m state \
--state NEW -j ACCEPT
$IPTABLES -A OUTPUT -o $INTERNET -p tcp --destination-port 443 -m state \
--state NEW -j ACCEPT
$IPTABLES -A OUTPUT -o $INTERNET -p tcp --destination-port 5190 -m state \
--state NEW -j ACCEPT
# Create a LOGDROP chain to log and drop packets
$IPTABLES -N LOGDROP
$IPTABLES -A LOGDROP -j LOG
$IPTABLES -A LOGDROP -j DROP
# Drop all other traffic
$IPTABLES -A INPUT -j LOGDROP
# Have these rules take effect when iptables is started
/sbin/service iptables save
The following iptables firewall is suited for a dual-homed Squid proxy server. ssh (TCP port 22), squid (TCP port 3128), and ICMP ECHO requests are allowed on the internal (LAN) interface.
Squid is configured to proxy ftp, http, https, and AOL Instant Messenger traffic. In addition, the server is running a caching/forwarding name server and time server and therefore requires therefore requires outgoing UDP port 123 (ntp) and TCP/UDP port 53 (dns).
#!/bin/sh
LAN="eth1"
INTERNET="eth0"
IPTABLES="/sbin/iptables"
# Kernel monitoring support
# More information:
# /usr/src/linux-`uname -r`/Documentation/networking/ip-sysctl.txt
# http://www.linuxgazette.com/book/view/1645
# http://www.spirit.com/Network/net0300.html
# Drop ICMP echo-request messages sent to broadcast or multicast addresses
echo 1 > /proc/sys/net/ipv4/icmp_echo_ignore_broadcasts
# Drop source routed packets
echo 0 > /proc/sys/net/ipv4/conf/all/accept_source_route
# Enable TCP SYN cookie protection from SYN floods
echo 1 > /proc/sys/net/ipv4/tcp_syncookies
# Don't accept ICMP redirect messages
echo 0 > /proc/sys/net/ipv4/conf/all/accept_redirects
# Don't send ICMP redirect messages
echo 0 > /proc/sys/net/ipv4/conf/all/send_redirects
# Enable source address spoofing protection
echo 1 > /proc/sys/net/ipv4/conf/all/rp_filter
# Log packets with impossible source addresses
echo 1 > /proc/sys/net/ipv4/conf/all/log_martians
# Needed for FTP (specifically, to allow incoming ftp-data connections)
/sbin/modprobe ip_conntrack_ftp
# Flush all chains
$IPTABLES --flush
# Allow unlimited traffic on the loopback interface
$IPTABLES -A INPUT -i lo -j ACCEPT
$IPTABLES -A OUTPUT -o lo -j ACCEPT
# Set default policies
$IPTABLES --policy INPUT DROP
$IPTABLES --policy OUTPUT DROP
$IPTABLES --policy FORWARD DROP
# Previously initiated and accepted exchanges bypass rule checking
$IPTABLES -A INPUT -m state --state ESTABLISHED,RELATED -j ACCEPT
$IPTABLES -A OUTPUT -m state --state ESTABLISHED,RELATED -j ACCEPT
# Allow incoming port 22 (ssh) connections on LAN interface
$IPTABLES -A INPUT -i $LAN -p tcp --destination-port 22 -m state \
--state NEW -j ACCEPT
# Allow incoming port 3128 (squid) connections on LAN interface
$IPTABLES -A INPUT -i $LAN -p tcp --destination-port 3128 -m state \
--state NEW -j ACCEPT
# Allow ICMP ECHO REQUESTS on LAN interface
$IPTABLES -A INPUT -i $LAN -p icmp --icmp-type echo-request -j ACCEPT
# Allow DNS resolution
$IPTABLES -A OUTPUT -o $INTERNET -p udp --destination-port 53 -m state \
--state NEW -j ACCEPT
$IPTABLES -A OUTPUT -o $INTERNET -p tcp --destination-port 53 -m state \
--state NEW -j ACCEPT
# Allow ntp synchronization
$IPTABLES -A OUTPUT -o $LAN -p udp --destination-port 123 -m state \
--state NEW -j ACCEPT
# Allow ssh on LAN interface
$IPTABLES -A OUTPUT -o $LAN -p tcp --destination-port 22 -m state \
--state NEW -j ACCEPT
# Allow Squid to proxy ftp, http, https, and AIM traffic
$IPTABLES -A OUTPUT -o $INTERNET -p tcp --destination-port 21 -m state \
--state NEW -j ACCEPT
$IPTABLES -A OUTPUT -o $INTERNET -p tcp --destination-port 80 -m state \
--state NEW -j ACCEPT
$IPTABLES -A OUTPUT -o $INTERNET -p tcp --destination-port 443 -m state \
--state NEW -j ACCEPT
$IPTABLES -A OUTPUT -o $INTERNET -p tcp --destination-port 5190 -m state \
--state NEW -j ACCEPT
# Create a LOGDROP chain to log and drop packets
$IPTABLES -N LOGDROP
$IPTABLES -A LOGDROP -j LOG
$IPTABLES -A LOGDROP -j DROP
# Drop all other traffic
$IPTABLES -A INPUT -j LOGDROP
# Have these rules take effect when iptables is started
/sbin/service iptables save
Multi homed iptables forword
Multi-homed iptables firewall
The following iptables firewall is suited for a dual-homed firewall. In this example, eth1 is the internal LAN interface and eth0 is the public Internet interface. All outbound and return traffic is allowed from both the internal LAN and the firewall itself. All incoming traffic originating from the Internet is dropped.
Note: for remote administration via ssh, I typically add a rule such as:
# Allow incoming ssh from work
/sbin/iptables -A INPUT -i eth0 -p tcp -s work_IP_address/32 --dport 22 -m state --state NEW -j ACCEPT
-----
#!/bin/sh
# Enable broadcast echo protection
echo 1 > /proc/sys/net/ipv4/icmp_echo_ignore_broadcasts
# Disable source routed packets
for f in /proc/sys/net/ipv4/conf/*/accept_source_route; do
echo 0 > $f
done
# Enable TCP SYN cookie protection
echo 1 > /proc/sys/net/ipv4/tcp_syncookies
# Disable ICMP Redirect acceptance
for f in /proc/sys/net/ipv4/conf/*/accept_redirects; do
echo 0 > $f
done
# Don't send Redirect messages
for f in /proc/sys/net/ipv4/conf/*/send_redirects; do
echo 0 > $f
done
# Drop spoofed packets coming in on an interface, which if replied to,
# would result in the reply going out a different interface.
for f in /proc/sys/net/ipv4/conf/*/rp_filter; do
echo 1 > $f
done
# Log packets with impossible addresses
for f in /proc/sys/net/ipv4/conf/*/log_martians; do
echo 1 > $f
done
# Flush existing rules on INPUT, OUTPUT, FORWARD chains and nat table
/sbin/iptables --flush
/sbin/iptables -t nat --flush
# Unlimited traffic on the loopback interface
/sbin/iptables -A INPUT -i lo -j ACCEPT
/sbin/iptables -A OUTPUT -o lo -j ACCEPT
# Set the default policy to drop
/sbin/iptables --policy INPUT DROP
/sbin/iptables --policy OUTPUT DROP
/sbin/iptables --policy FORWARD DROP
/sbin/iptables -t nat --policy PREROUTING ACCEPT
/sbin/iptables -t nat --policy OUTPUT ACCEPT
/sbin/iptables -t nat --policy POSTROUTING ACCEPT
# Drop all invalid TCP state combinations
# First list of TCP state flags lists the bits to be tested
# Second list of TCP state flags lists the bits that must be set to match test
# All of the bits are cleared
/sbin/iptables -A INPUT -p tcp --tcp-flags ALL NONE -j DROP
# SYN and FIN are both set
/sbin/iptables -A INPUT -p tcp --tcp-flags SYN,FIN SYN,FIN -j DROP
# SYN and RST are both set
/sbin/iptables -A INPUT -p tcp --tcp-flags SYN,RST SYN,RST -j DROP
# FIN and RST are both set
/sbin/iptables -A INPUT -p tcp --tcp-flags FIN,RST FIN,RST -j DROP
# FIN is set without the expected accompanying ACK
/sbin/iptables -A INPUT -p tcp --tcp-flags ACK,FIN FIN -j DROP
# PSH is set without the expected accompanying ACK
/sbin/iptables -A INPUT -p tcp --tcp-flags ACK,PSH PSH -j DROP
# URG is set without the expected accompanying ACK
/sbin/iptables -A INPUT -p tcp --tcp-flags ACK,URG URG -j DROP
# Masquerade everything out eth0; used for dynamic IPs
/sbin/iptables -t nat -A POSTROUTING -o eth0 -j MASQUERADE
# Allow all outbound connections from LAN (eth1) to Internet (eth0)
# Allow only return traffic from those connections
/sbin/iptables -A FORWARD -i eth1 -o eth0 -m state --state NEW,ESTABLISHED,RELATED -j ACCEPT
/sbin/iptables -A FORWARD -i eth0 -o eth1 -m state --state ESTABLISHED,RELATED -j ACCEPT
# Allow unlimited outbound and return traffic from firewall
/sbin/iptables -A INPUT -m state --state ESTABLISHED,RELATED -j ACCEPT
/sbin/iptables -A OUTPUT -o eth0 -m state --state NEW,ESTABLISHED,RELATED -j ACCEPT
/sbin/iptables -A OUTPUT -o eth1 -m state --state NEW,ESTABLISHED,RELATED -j ACCEPT
# Activate IP forwarding
/bin/echo 1 > /proc/sys/net/ipv4/ip_forward
# Save iptables rules
/sbin/service iptables save
The following iptables firewall is suited for a dual-homed firewall. In this example, eth1 is the internal LAN interface and eth0 is the public Internet interface. All outbound and return traffic is allowed from both the internal LAN and the firewall itself. All incoming traffic originating from the Internet is dropped.
Note: for remote administration via ssh, I typically add a rule such as:
# Allow incoming ssh from work
/sbin/iptables -A INPUT -i eth0 -p tcp -s work_IP_address/32 --dport 22 -m state --state NEW -j ACCEPT
-----
#!/bin/sh
# Enable broadcast echo protection
echo 1 > /proc/sys/net/ipv4/icmp_echo_ignore_broadcasts
# Disable source routed packets
for f in /proc/sys/net/ipv4/conf/*/accept_source_route; do
echo 0 > $f
done
# Enable TCP SYN cookie protection
echo 1 > /proc/sys/net/ipv4/tcp_syncookies
# Disable ICMP Redirect acceptance
for f in /proc/sys/net/ipv4/conf/*/accept_redirects; do
echo 0 > $f
done
# Don't send Redirect messages
for f in /proc/sys/net/ipv4/conf/*/send_redirects; do
echo 0 > $f
done
# Drop spoofed packets coming in on an interface, which if replied to,
# would result in the reply going out a different interface.
for f in /proc/sys/net/ipv4/conf/*/rp_filter; do
echo 1 > $f
done
# Log packets with impossible addresses
for f in /proc/sys/net/ipv4/conf/*/log_martians; do
echo 1 > $f
done
# Flush existing rules on INPUT, OUTPUT, FORWARD chains and nat table
/sbin/iptables --flush
/sbin/iptables -t nat --flush
# Unlimited traffic on the loopback interface
/sbin/iptables -A INPUT -i lo -j ACCEPT
/sbin/iptables -A OUTPUT -o lo -j ACCEPT
# Set the default policy to drop
/sbin/iptables --policy INPUT DROP
/sbin/iptables --policy OUTPUT DROP
/sbin/iptables --policy FORWARD DROP
/sbin/iptables -t nat --policy PREROUTING ACCEPT
/sbin/iptables -t nat --policy OUTPUT ACCEPT
/sbin/iptables -t nat --policy POSTROUTING ACCEPT
# Drop all invalid TCP state combinations
# First list of TCP state flags lists the bits to be tested
# Second list of TCP state flags lists the bits that must be set to match test
# All of the bits are cleared
/sbin/iptables -A INPUT -p tcp --tcp-flags ALL NONE -j DROP
# SYN and FIN are both set
/sbin/iptables -A INPUT -p tcp --tcp-flags SYN,FIN SYN,FIN -j DROP
# SYN and RST are both set
/sbin/iptables -A INPUT -p tcp --tcp-flags SYN,RST SYN,RST -j DROP
# FIN and RST are both set
/sbin/iptables -A INPUT -p tcp --tcp-flags FIN,RST FIN,RST -j DROP
# FIN is set without the expected accompanying ACK
/sbin/iptables -A INPUT -p tcp --tcp-flags ACK,FIN FIN -j DROP
# PSH is set without the expected accompanying ACK
/sbin/iptables -A INPUT -p tcp --tcp-flags ACK,PSH PSH -j DROP
# URG is set without the expected accompanying ACK
/sbin/iptables -A INPUT -p tcp --tcp-flags ACK,URG URG -j DROP
# Masquerade everything out eth0; used for dynamic IPs
/sbin/iptables -t nat -A POSTROUTING -o eth0 -j MASQUERADE
# Allow all outbound connections from LAN (eth1) to Internet (eth0)
# Allow only return traffic from those connections
/sbin/iptables -A FORWARD -i eth1 -o eth0 -m state --state NEW,ESTABLISHED,RELATED -j ACCEPT
/sbin/iptables -A FORWARD -i eth0 -o eth1 -m state --state ESTABLISHED,RELATED -j ACCEPT
# Allow unlimited outbound and return traffic from firewall
/sbin/iptables -A INPUT -m state --state ESTABLISHED,RELATED -j ACCEPT
/sbin/iptables -A OUTPUT -o eth0 -m state --state NEW,ESTABLISHED,RELATED -j ACCEPT
/sbin/iptables -A OUTPUT -o eth1 -m state --state NEW,ESTABLISHED,RELATED -j ACCEPT
# Activate IP forwarding
/bin/echo 1 > /proc/sys/net/ipv4/ip_forward
# Save iptables rules
/sbin/service iptables save
Iptables
Check Point NG problems with EDNS
We recently ran into problems with Check Point NG FP3 and extremely slow DNS resolution. Recursive DNS queries that normally took less than a second to resolve took over 20 seconds. Our name server is running BIND 9.2.3, although this problem may affect other name servers using EDNS.
The problem was that Check Point NG was configured with "SmartDefense" for DNS UDP packets, effectively blocking any DNS UDP request with a UDP payload greater than 512 bytes. With BIND 9.2.3, DNS queries are larger than 512 bytes, as they contain an "EDNS0 option."
Here is a packet capture for a DNS A record query for www.chadangerer.com:
15:26:13.989203 192.168.1.100.54859 > 216.119.106.2.53: 60974 [1au] A? www.chadangerer.com. (48) (DF)
Note the "[1au]". From the tcpdump man page:
A few anomalies are checked and may result in extra fields enclosed in square brackets: If a query contains an answer, authority records or additional records section, ancount, nscount, or arcount are printed as `[na]', `[nn]' or `[nau]' where n is the appropriate count.
In this case, the DNS query contained one additional record. Capturing the packets in raw format (tcpdump -w) and loading the data into Wireshark, we see that the additional record is an "EDNS0 option" with a UDP payload size of 2048:
Additional records
: type OPT, class unknown
Name:
Type: EDNS0 option
UDP payload size: 2048
Higher bits in extended RCODE: 0x0
EDNS0 version: 0
Z: 0x0
Data length: 0
Data
BIND attempts to resolve the DNS A two more times with the EDNS0 option:
15:26:15.990202 192.168.1.100.54859 > 216.119.106.3.53: 8978 [1au] A? www.chadangerer.com. (48) (DF)
15:26:18.000718 192.168.1.100.54859 > 216.119.106.2.53: 20148 [1au] A? www.chadangerer.com. (48) (DF)
Finally, BIND sends the DNS A record request without the EDNS0 option. The DNS reply is immediately returned.
15:26:20.010329 192.168.1.100.54859 > 216.119.106.3.53: 21909 A? www.chadangerer.com. (37) (DF)
15:26:20.179621 216.119.106.3.53 > 192.168.1.100.54859: 21909* 1/0/0 A 65.110.86.239 (53)
The solution to the problem was to disable "SmartDefense" for DNS UDP packets.
More information:
http://lists.virus.org/fw1-0305/msg00433.html
We recently ran into problems with Check Point NG FP3 and extremely slow DNS resolution. Recursive DNS queries that normally took less than a second to resolve took over 20 seconds. Our name server is running BIND 9.2.3, although this problem may affect other name servers using EDNS.
The problem was that Check Point NG was configured with "SmartDefense" for DNS UDP packets, effectively blocking any DNS UDP request with a UDP payload greater than 512 bytes. With BIND 9.2.3, DNS queries are larger than 512 bytes, as they contain an "EDNS0 option."
Here is a packet capture for a DNS A record query for www.chadangerer.com:
15:26:13.989203 192.168.1.100.54859 > 216.119.106.2.53: 60974 [1au] A? www.chadangerer.com. (48) (DF)
Note the "[1au]". From the tcpdump man page:
A few anomalies are checked and may result in extra fields enclosed in square brackets: If a query contains an answer, authority records or additional records section, ancount, nscount, or arcount are printed as `[na]', `[nn]' or `[nau]' where n is the appropriate count.
In this case, the DNS query contained one additional record. Capturing the packets in raw format (tcpdump -w) and loading the data into Wireshark, we see that the additional record is an "EDNS0 option" with a UDP payload size of 2048:
Additional records
Name:
Type: EDNS0 option
UDP payload size: 2048
Higher bits in extended RCODE: 0x0
EDNS0 version: 0
Z: 0x0
Data length: 0
Data
BIND attempts to resolve the DNS A two more times with the EDNS0 option:
15:26:15.990202 192.168.1.100.54859 > 216.119.106.3.53: 8978 [1au] A? www.chadangerer.com. (48) (DF)
15:26:18.000718 192.168.1.100.54859 > 216.119.106.2.53: 20148 [1au] A? www.chadangerer.com. (48) (DF)
Finally, BIND sends the DNS A record request without the EDNS0 option. The DNS reply is immediately returned.
15:26:20.010329 192.168.1.100.54859 > 216.119.106.3.53: 21909 A? www.chadangerer.com. (37) (DF)
15:26:20.179621 216.119.106.3.53 > 192.168.1.100.54859: 21909* 1/0/0 A 65.110.86.239 (53)
The solution to the problem was to disable "SmartDefense" for DNS UDP packets.
More information:
http://lists.virus.org/fw1-0305/msg00433.html
Iptables
Check Point FireWall-1 rule check script
I wrote the following script to count the number of times each rule is matched in our CheckPoint FireWall-1 security policy. The script is run once per week before the Check Point FireWall-1 logs are rotated.
You can optimize the rulebase by moving the most frequently accessed rules to the top of the security policy; the script can also help identify rules that are no longer used.
#!/bin/sh
# Variables
# TMP_OUTPUT is the file to store temporary output
# RECIPIENTS is a list of email recipients
TMP_OUTPUT=/tmp/fw_rule_check.tmp
RECIPIENTS=user@example.com
# Remove the temporary output file if it exists
[ -f $TMP_OUTPUT ] && rm $TMP_OUTPUT
/usr/bin/echo "Starting time: `date`\n" >> $TMP_OUTPUT
/usr/bin/echo "Rule\tCount" >> $TMP_OUTPUT
/usr/bin/echo "----\t-----" >> $TMP_OUTPUT
# For every line returned by "fw log," count the rule.
# The "rule (number)" is not in the same place on every line, so Perl
# is used to extract the rule.
/opt/CKPfw/bin/fw log | /usr/bin/perl -ne 'print "$1\n" if /rule\s(\d+)/' | \
/usr/bin/sort -n | /usr/bin/uniq -c | /usr/bin/awk '{print $2 "\t" $1}' >> $TMP_OUTPUT
/usr/bin/echo "\nEnding time: `date`" >> $TMP_OUTPUT
/usr/bin/mailx -s "Firewall rule check" $RECIPIENTS < $TMP_OUTPUT
rm $TMP_OUTPUT
Example output:
Starting time: Sat Dec 7 22:00:00 CST 2002
Rule Count
---- -----
0 147262
2 1
4 886295
6 19650
7 13993
8 13160
11 142
12 3741
14 5114
20 8
28 33
40 1878
41 505
52 162
53 3
54 3
56 40
57 88
58 28502
59 258141
60 106993
Ending time: Sun Dec 8 02:02:24 CST 2002
I wrote the following script to count the number of times each rule is matched in our CheckPoint FireWall-1 security policy. The script is run once per week before the Check Point FireWall-1 logs are rotated.
You can optimize the rulebase by moving the most frequently accessed rules to the top of the security policy; the script can also help identify rules that are no longer used.
#!/bin/sh
# Variables
# TMP_OUTPUT is the file to store temporary output
# RECIPIENTS is a list of email recipients
TMP_OUTPUT=/tmp/fw_rule_check.tmp
RECIPIENTS=user@example.com
# Remove the temporary output file if it exists
[ -f $TMP_OUTPUT ] && rm $TMP_OUTPUT
/usr/bin/echo "Starting time: `date`\n" >> $TMP_OUTPUT
/usr/bin/echo "Rule\tCount" >> $TMP_OUTPUT
/usr/bin/echo "----\t-----" >> $TMP_OUTPUT
# For every line returned by "fw log," count the rule.
# The "rule (number)" is not in the same place on every line, so Perl
# is used to extract the rule.
/opt/CKPfw/bin/fw log | /usr/bin/perl -ne 'print "$1\n" if /rule\s(\d+)/' | \
/usr/bin/sort -n | /usr/bin/uniq -c | /usr/bin/awk '{print $2 "\t" $1}' >> $TMP_OUTPUT
/usr/bin/echo "\nEnding time: `date`" >> $TMP_OUTPUT
/usr/bin/mailx -s "Firewall rule check" $RECIPIENTS < $TMP_OUTPUT
rm $TMP_OUTPUT
Example output:
Starting time: Sat Dec 7 22:00:00 CST 2002
Rule Count
---- -----
0 147262
2 1
4 886295
6 19650
7 13993
8 13160
11 142
12 3741
14 5114
20 8
28 33
40 1878
41 505
52 162
53 3
54 3
56 40
57 88
58 28502
59 258141
60 106993
Ending time: Sun Dec 8 02:02:24 CST 2002
Iptables
Check Point FireWall-1 Management Console port forwarding
Check Point FireWall-1 only allows IP addresses in the access list $FWDIR/conf/gui-clients to use the Management Console. Unfortunately, this file does not accept ranges of IP addresses. What if you want to connect to the Management Console from different locations without having to manually edit this file?
The best (and most secure) way to connect to the Management Console is to use ssh port forwarding with an ssh client like PuTTY. Instructions on how to configure the ssh tunnel using OpenSSH are also listed below. Using ssh port forwarding of course requires an ssh server installed on your Management Console.
Using ssh port forwarding, the Management Console believes your network traffic is originating from localhost, which is always allowed to connect to the Management Console. In this example, I'll use firewall to as the hostname of the Management Console.
PuTTY configuration:
1. Click on SSH/Tunnels
2. Enter 258 in the Source port dialog box.
3. Enter firewall:258 in the Destination dialog box.
4. Leave the Local radio button checked.
5. Click Add.
6. Click Session and connect to firewall with the ssh protocol.
7. After entering your username and password through PuTTY, startup the Check Point FireWall-1 Management Console. Your Check Point Username: and Password: will be the same. However, you must enter localhost for Management Server:
OpenSSH configuration
If you using OpenSSH, make sure you have the environment variable DISPLAY=localhost:0 set, and issue the following command:
ssh -L 258:firewall:258 UNIX_user_ID@firewall
You should be now be able to access the Check Point FireWall-1 Management Console from anywhere (assuming you can reach the Management Console via TCP ports 22 and 258).
Check Point FireWall-1 only allows IP addresses in the access list $FWDIR/conf/gui-clients to use the Management Console. Unfortunately, this file does not accept ranges of IP addresses. What if you want to connect to the Management Console from different locations without having to manually edit this file?
The best (and most secure) way to connect to the Management Console is to use ssh port forwarding with an ssh client like PuTTY. Instructions on how to configure the ssh tunnel using OpenSSH are also listed below. Using ssh port forwarding of course requires an ssh server installed on your Management Console.
Using ssh port forwarding, the Management Console believes your network traffic is originating from localhost, which is always allowed to connect to the Management Console. In this example, I'll use firewall to as the hostname of the Management Console.
PuTTY configuration:
1. Click on SSH/Tunnels
2. Enter 258 in the Source port dialog box.
3. Enter firewall:258 in the Destination dialog box.
4. Leave the Local radio button checked.
5. Click Add.
6. Click Session and connect to firewall with the ssh protocol.
7. After entering your username and password through PuTTY, startup the Check Point FireWall-1 Management Console. Your Check Point Username: and Password: will be the same. However, you must enter localhost for Management Server:
OpenSSH configuration
If you using OpenSSH, make sure you have the environment variable DISPLAY=localhost:0 set, and issue the following command:
ssh -L 258:firewall:258 UNIX_user_ID@firewall
You should be now be able to access the Check Point FireWall-1 Management Console from anywhere (assuming you can reach the Management Console via TCP ports 22 and 258).
Iptables
Here are various notes I have on Check Point FireWall-1 4.0 that are not detailed enough to warrant their own Web page.
Adding a Management Console user
# ./fwm -a user
Deleting Management Console user
# ./fwm -r user
Displaying all Management Console users and associated permissions
# ./fwm -p
Manually loading a security policy
To manually install a CheckPoint FireWall security policy, use the "fw load" command, followed by the name of the policy, and the server destination.
Example:
# ./fw load /var/opt/CKPfw/conf/policy_file.W hostname
Empty security policy
If you see an empty security policy (i.e. you only see Standard.W) when using the Security Policy GUI, the $FWDIR/conf/rulebases.fws file may be corrupt. This could be caused by making a manual policy/object modification without coordinating the change in rulebases.fws. It is of course best to make modifications only through the Security Policy GUI.
More information:
http://groups.google.com/groups?hl=en&threadm=6etvds%243bq%241%40mailhost2.dtc.co.jp&rnum=1&prev=/groups%3Fq%3Dcheckpoint%2Bempty%2Bsecurity%2Bpolicy%26hl%3Den%26rnum%3D1%26selm%3D6etvds%25243bq%25241%2540mailhost2.dtc.co.jp
Adding a Management Console user
# ./fwm -a user
Deleting Management Console user
# ./fwm -r user
Displaying all Management Console users and associated permissions
# ./fwm -p
Manually loading a security policy
To manually install a CheckPoint FireWall security policy, use the "fw load" command, followed by the name of the policy, and the server destination.
Example:
# ./fw load /var/opt/CKPfw/conf/policy_file.W hostname
Empty security policy
If you see an empty security policy (i.e. you only see Standard.W) when using the Security Policy GUI, the $FWDIR/conf/rulebases.fws file may be corrupt. This could be caused by making a manual policy/object modification without coordinating the change in rulebases.fws. It is of course best to make modifications only through the Security Policy GUI.
More information:
http://groups.google.com/groups?hl=en&threadm=6etvds%243bq%241%40mailhost2.dtc.co.jp&rnum=1&prev=/groups%3Fq%3Dcheckpoint%2Bempty%2Bsecurity%2Bpolicy%26hl%3Den%26rnum%3D1%26selm%3D6etvds%25243bq%25241%2540mailhost2.dtc.co.jp
Iptables
"Server Not Responding" errors in CKPfw Security Policy
Check Point's Security Policy loads all objects (in objects.C) and rulebases (rulebases.fws) when starting up. When these files become large, the time to load may exceed the default 25 second timeout value resulting in "Server Not Responding" or "Incorrect reply from server (seq or subject mismatch) messages.
You may either:
1. Reduce the number of rulebases loaded (recommended). Back up the existing rulebases.fws file, and create a new rulebases.fws with one or more of your rulebases.
mv $FW_DIR/conf/rulebases.fws $FW_DIR/conf/rulebases.fws.`date +%m.%d.%y`
$FW_DIR/bin/fwm -g $FW_DIR/conf/rulebase.W
Note: you probably want to add more than one rulebase to make it easy to revert to previous rulebases.
2. Increase the default 25 second timeout value:
ex. SERVER_TIMEOUT 45 $FW_DIR/bin/fwpolicy &
Check Point's Security Policy loads all objects (in objects.C) and rulebases (rulebases.fws) when starting up. When these files become large, the time to load may exceed the default 25 second timeout value resulting in "Server Not Responding" or "Incorrect reply from server (seq or subject mismatch) messages.
You may either:
1. Reduce the number of rulebases loaded (recommended). Back up the existing rulebases.fws file, and create a new rulebases.fws with one or more of your rulebases.
mv $FW_DIR/conf/rulebases.fws $FW_DIR/conf/rulebases.fws.`date +%m.%d.%y`
$FW_DIR/bin/fwm -g $FW_DIR/conf/rulebase.W
Note: you probably want to add more than one rulebase to make it easy to revert to previous rulebases.
2. Increase the default 25 second timeout value:
ex. SERVER_TIMEOUT 45 $FW_DIR/bin/fwpolicy &
Iptables
Example iptables firewall
The following is an example iptables firewall that allows incoming ssh connections from an individual IP address (192.168.1.100), allows all outbound traffic, and uses stateful inspection.
This iptables firewall is suited for a single-homed firewall with support for remote ssh administration. For personal use, you may not want egress (outbound) filtering of network traffic. Of course, a multi-homed corporate firewall should employ egress filtering as well as ingress (inbound) filtering.
The script was developed based on information in Robert Ziegler's Linux Firewalls book. More information on iptables is available at http://www.linux-firewall-tools.com/linux/
#!/bin/sh
# Kernel monitoring support
# More information:
# /usr/src/linux-`uname -r`/Documentation/networking/ip-sysctl.txt
# http://www.linuxgazette.com/book/view/1645
# http://www.spirit.com/Network/net0300.html
# Drop ICMP echo-request messages sent to broadcast or multicast addresses
echo 1 > /proc/sys/net/ipv4/icmp_echo_ignore_broadcasts
# Drop source routed packets
echo 0 > /proc/sys/net/ipv4/conf/all/accept_source_route
# Enable TCP SYN cookie protection from SYN floods
echo 1 > /proc/sys/net/ipv4/tcp_syncookies
# Don't accept ICMP redirect messages
echo 0 > /proc/sys/net/ipv4/conf/all/accept_redirects
# Don't send ICMP redirect messages
echo 0 > /proc/sys/net/ipv4/conf/all/send_redirects
# Enable source address spoofing protection
echo 1 > /proc/sys/net/ipv4/conf/all/rp_filter
# Log packets with impossible source addresses
echo 1 > /proc/sys/net/ipv4/conf/all/log_martians
# Flush all chains
/sbin/iptables --flush
# Allow unlimited traffic on the loopback interface
/sbin/iptables -A INPUT -i lo -j ACCEPT
/sbin/iptables -A OUTPUT -o lo -j ACCEPT
# Set default policies
/sbin/iptables --policy INPUT DROP
/sbin/iptables --policy OUTPUT DROP
/sbin/iptables --policy FORWARD DROP
# Previously initiated and accepted exchanges bypass rule checking
# Allow unlimited outbound traffic
/sbin/iptables -A INPUT -m state --state ESTABLISHED,RELATED -j ACCEPT
/sbin/iptables -A OUTPUT -m state --state NEW,ESTABLISHED,RELATED -j ACCEPT
# Allow incoming TCP port 22 (ssh) traffic from office
/sbin/iptables -A INPUT -p tcp -s 192.168.1.100 --dport 22 -m state --state NEW -j ACCEPT
# Drop all other traffic
/sbin/iptables -A INPUT -j DROP
# Have these rules take effect when iptables is started
/sbin/service iptables save
That is the end of the original script.
If you want to make a syslog entry of dropped packets, change:
# Drop all other traffic
/sbin/iptables -A INPUT -j DROP
To:
# Create a LOGDROP chain to log and drop packets
/sbin/iptables -N LOGDROP
/sbin/iptables -A LOGDROP -j LOG
/sbin/iptables -A LOGDROP -j DROP
# Drop all other traffic
/sbin/iptables -A INPUT -j LOGDROP
You may also want to configure the --log-level to log dropped packets to a separate file instead of /var/log/messages:
# Drop all other traffic
/sbin/iptables -A INPUT -j LOGDROP --log-level debug
/etc/syslog.conf change:
# Send iptables LOGDROPs to /var/log/iptables
kern.=debug /var/log/iptables
Reload the syslogd service for the change to take effect.
/sbin/service syslog reload
If you do not want to allow incoming ssh, remove:
/sbin/iptables -A INPUT -p tcp -s 192.168.1.100/32 --dport 22 -m state --state NEW -j అచ్సుప్ట్
Subscribe to:
Posts (Atom)